User-defined virtual interaction space and manipulation of virtual cameras in the interaction space

ABSTRACT

The technology disclosed relates to creating user-defined interaction spaces and modalities in a three dimensional (3D) sensor space in response to control gestures. It also relates to controlling virtual cameras in the sensor space using control gestures and manipulating controls of the virtual cameras through the control gestures. In particular, it relates to defining one or more spatial attributes of the interaction spaces and modalities in response to one or more gesture parameters of the control gesture. It also particularly relates to defining one or more visual parameters of a virtual camera in response to one or more gesture parameters of the control gesture.

PRIORITY DATA

This application claims the benefit of U.S. Provisional PatentApplication No. 61/916,790, entitled, “USER-DEFINED VIRTUAL INTERACTIONSPACE AND MANIPULATION OF VIRTUAL CAMERAS IN THE INTERACTION SPACE,”filed on Dec. 16, 2013 (Attorney Docket No. LEAP 1020-1). Theprovisional application is hereby incorporated by reference for allpurposes.

INCORPORATIONS

Materials incorporated by reference in this filing include thefollowing:

“Contactless Cursor Control Using Free-Space Motion Detection,” U.S.Prov. App. No. 61/825,480, filed 20 May 2013 (Attorney Docket No. LEAP1001-1),

“Predictive Information for Free Space Gesture Control andCommunication,” U.S. Prov. App. No. 61/871,790, filed 29 Aug. 2013(Attorney Docket No. LEAP 1006-1/LPM-1006PR),

“Predictive Information for Free-space Gesture Control andCommunication,” U.S. Prov. App. No. 61/873,758, filed 04 Sep. 2013(Attorney Docket No. LEAP 1007-1/LMP-1007PR),

“Predictive Information For Free Space Gesture Control AndCommunication,” U.S. Non. Prov. App. No. 14/474,077, filed 29 Aug. 2014(Attorney Docket No. LEAP 1007-2/LPM-1007US),

“Velocity Field Interaction for Free Space Gesture Interface andControl,” U.S. Prov. App. No. 61/891,880, filed 16 Oct. 2013 (AttorneyDocket No. LEAP 1008-1/1009APR),

“Velocity Field Interaction For Free Space Gesture Interface AndControl,” U.S. Non. Prov. application No. 14/516,493, filed 16 Oct. 2014(Attorney Docket No. LEAP 1008-2/LPM-1008US),

“Virtual Interactions For Machine Control,” U.S. Prov. App. No.61/897,186, filed 29 Oct. 2013, (Attorney Docket No. LEAP1016-1/LPM-1016PR),

“Virtual Interactions For Machine Control,” U.S. Non Prov. applicationNo. 14/527,742, filed 29 Oct. 2014, (Attorney Docket No. LEAP1016-2/LPM-1016US),

“Interactions With Virtual Objects For Machine Control,” U.S. Prov. App.No. 61/898,464, filed 31 Oct. 2013, (Attorney Docket No. LEAP1017-1/LPM-1017PR),

“Interactions With Virtual Objects For Machine Control,” U.S. Non Prov.application No. 14/530,364, filed 31 Oct. 2014, (Attorney Docket No.LEAP 1017-2/LPM-1017US),

“Improving Predictive Information For Free Space Gesture Control AndCommunication,” US Prov. App. No. 61/898,462, filed 31 Oct. 2013,(Attorney Docket No. LEAP 1018-1/LPM-1018PR),

“Improving Predictive Information For Free Space Gesture Control AndCommunication,” US Non Prov. application No. 14/530,690, filed 31 Oct.2014, (Attorney Docket No. LEAP 1018-2/LPM-1018US),

“Interaction Strength Using Virtual Objects For Machine Control,” U.S.Prov. App. No. 61/905,103, filed 15 Nov. 2013, (Attorney Docket No. LEAP1021-1/LPM-1021PR),

“Interaction Strength Using Virtual Objects For Machine Control,” U.S.Non Prov. application No. 14/541,078, filed 13 Nov. 2014, (AttorneyDocket No. LEAP 1021-2/LPM-1021US),

“Vehicle Motion Sensory Control,” U.S. Prov. App. No. 62/005,981, filed30 May 2014, (Attorney Docket No. LEAP 1052-1/LPM-1052PR),

“Free-Space User Interface And Control Using Virtual Constructs,” U.S.Non. Prov. application No. 14/154,730, filed 14 Jan. 2014 (AttorneyDocket No. LEAP 1068-2/LPM-033US),

“Free-Space User Interface And Control Using Virtual Constructs,” U.S.Prov. App. No. 61/873,351, filed 03 September 2013 (Attorney Docket No.LPM-033PR3/7315741001),

“Free-Space User Interface And Control Using Virtual Constructs,” USProv. App. No. 61/877,641, filed 13 Sep. 2013 (Attorney Docket No.LPM-033PR4),

“Systems And Methods For Machine Control,” U.S. Non. Prov. applicationNo. 14/280,018, filed 16 May 2014 (Attorney Docket No.LPM-0001CP3/7312204003),

“Dynamic, Free-Space User Interactions For Machine Control,” U.S. Non.Prov. application No. 14/155,722, filed 15 Jan. 2014 (Attorney DocketNo. LPM-033CP/7315744001),

“Interactive Training Recognition of Free Space Gestures for Interfaceand Control,” U.S. Prov. App. No. 61/872,538, filed 30 Aug. 2013(Attorney Docket No. LPM-013GPR/7312701007),

“Methods and systems for identifying position and shape of objects inthree-dimensional space,” U.S. Prov. App. No. 61/587,554, filed 17 Jan.2012 (Attorney Docket No. PA5663PRV),

“Systems and methods for capturing motion in three-dimensional space,”U.S. Prov. App. No. 61/724,091, filed 8 Nov. 2012 (Attorney Docket No.LPM-001PR2/7312201010),

“Non-tactile interface systems and methods,” U.S. Prov. App. No.61/816,487, filed 26 Apr. 2013 (Attorney Docket No.LPM-028PR/7313971001),

“Dynamic user interactions for display control,” U.S. Prov. App. No.61/752,725, filed 15 Jan. 2013 (Attorney Docket No.LPM-013APR/7312701001),

“Motion capture using cross-sections of an object,” U.S. application No.13/414,485, filed 7 Mar. 2012 (Attorney Docket No. LEAP1006-7/LPM-1006US),

“System and methods for capturing motion in three-dimensional space,”U.S. application No. 13/742,953, filed 16 Jan. 2013 (Attorney Docket No.LPM-001CP2/7312204002),

“User-Defined Virtual Interaction Space and Manipulation of VirtualCameras with Vectors,” U.S. application No. 14/572,690, filed 16 Dec.2014 (Atty. Docket No. LEAP 1020-3/LPM-1020US2), and

“User-Defined Virtual Interaction Space and Manipulation of VirtualConfiguration,” U.S. application No. 14/572,704, filed 16 Dec. 2014(Atty. Docket No. LEAP 1020-4/LPM-1020US3).

TECHNICAL FIELD

The technology described relates to machine user interfaces, and morespecifically to the use of virtual objects as user input to machines.

DISCUSSION

Conventional machine interfaces are in common daily use. Every day,millions of users type their commands, click their computer mouse andhope for the best.

Unfortunately, however, these types of interfaces are very limited.

Therefore, what is needed is a remedy to this and other shortcomings ofthe traditional machine interface approaches.

SUMMARY

Aspects of the systems and methods described provide for improvedcontrol of machines or other computing resources based at least in parton determining whether positions and/or motions of an object (e.g.,hand, tool, hand and tool combinations, other detectable objects orcombinations thereof) might be interpreted as an interaction with one ormore virtual objects, controls or content. Implementations can enablemodeling of physical objects, created objects and interactions withvarious combinations thereof for machine control or other purposes.

In one implementation, a method is described for creating user-definedinterface modalities in a three dimensional (3D) sensor space. Themethod includes detecting a control gesture of a control object,calculating gesture parameters of the control gesture that was detected,and defining spatial attributes of an interaction modality in the 3Dsensor space responsive to the gesture parameters of the controlgesture. The gesture parameters include at least length and width of thecontrol gesture. The gesture parameters also can include at leaststructure, scale, orientation, or density of the control object. Thespatial attributes include at least height and width of an interactionspace. The spatial attributes can also include at least numerosity ofelements in the interaction modality.

Aspects of this implementation that are described below are not repeatedfor each different implementation, for the sake of brevity. It should beunderstood

A context-setting control gesture can be detected, which identifies acontext for interpreting a subsequent control gesture that definesspatial attributes of the interaction modality. The context-settingcontrol gesture can be a voice, visual, or device command. Subsequentcontrol gestures can apply to an entire interaction space. Subsequentcontrol gestures can also apply to an element of the interaction space.

Context-aware elements of the interaction modality can be created thatautomatically interpret a context-setting control gesture and subsequentcontrol gestures to define spatial attributes of the interactionmodality. The control gesture can be a stroke of a user appendage. Inanother implementation, the control object is a detectable object andthe control gesture defines a collection of continuous points that haveat least one parameter in common within a threshold deviation. Thethreshold deviation can be determined by a variation in angle alongvelocity vectors that are continuous in time. The control gesture canalso be a circular sweep that defines a collection of points within aradial distance to a fixed point.

In some implementations, a method is described for creating user-definedinterface modalities in a 3D sensor space using a stroke of a controlobject that manipulate controls in a physical interaction space. Themethod includes detecting a vertical sweep of a control objectresponsive to a first control gesture in a 3D sensor space, defining avertical extent of a virtual interaction space in proportion to lengthof vertical sweep of the control object, detecting a horizontal sweep ofthe control object responsive to a second control gesture in the 3Dsensor space, defining a horizontal extent of the virtual interactionspace in proportion to width of horizontal sweep of the control object,and manipulating controls in a physical interaction space bysuperimposing the virtual interaction space on the physical interactionspace responsive to the vertical extent and horizontal extent.

A method can be described for creating user-defined interface modalitiesin a 3D sensor space using a stroke of a control object that manipulatecontrols in a synthetic interaction space. The method includes detectinga vertical sweep of a control object responsive to a first controlgesture in a 3D sensor space, defining a vertical extent of a virtualinteraction space in proportion to length of vertical sweep of thecontrol object, detecting a horizontal sweep of the control objectresponsive to a second control gesture in the 3D sensor space, defininga horizontal extent of the virtual interaction space in proportion towidth of horizontal sweep of the control object, and manipulatingcontrols in a synthetic interaction space by linking the virtualinteraction space to an image responsive to the vertical extent andhorizontal extent

A method also can be described for creating user-defined interfacemodalities in a 3D sensor space using a circular sweep of a controlobject that manipulate controls in a physical interaction space. Themethod includes circular sweep of a control object responsive to acontrol gesture in a 3D sensor space, calculating a radius of thecircular sweep based on a found point that is equidistant to a pluralityof points defined on contour of the control gesture, constructing aradial-based virtual interaction modality in the 3D sensor space that isin proportion to the radius of the circular sweep, and manipulatingcontrols in a physical interaction space by superimposing theradial-based virtual interaction modality on the physical interactionspace responsive to the circular sweep.

A method can further be described for creating user-defined interfacemodalities in a 3D sensor space using a circular sweep of a controlobject that manipulate controls in a synthetic interaction space. Themethod includes circular sweep of a control object responsive to acontrol gesture in a 3D sensor space, calculating a radius of thecircular sweep based on a found point that is equidistant to a pluralityof points defined on contour of the control gesture, constructing aradial-based virtual interaction modality in the 3D sensor space that isin proportion to the radius of the circular sweep, and manipulatingcontrols in a synthetic interaction space by linking the radial-basedvirtual interaction modality to an image responsive to the verticalextent and horizontal extent.

In some implementations, a method is described for creating user-definedinterface modalities in a 3D sensor space using lateral outward movementof control objects. The method includes identifying a pair of startingpoints in respective centers of two control objects that are detected ina 3D sensor space, wherein the pair of starting points are fixeddistance apart, detecting an outward expanding movement of the controlobjects in the 3D sensor space, identifying a pair of resting points inrespective centers of the two control objects when the control objectscome to rest, defining a horizontal extent of a virtual interactionspace in proportion to distance between the starting points and theresting points, defining a vertical extent of the virtual interactionspace in proportion to width of the control objects, and presenting theinteraction space responsive to the vertical extent and horizontalextent. In one implementation, the two control objects are two userappendages.

A method can be described for creating user-defined interface modalitiesin a 3D sensor space using lateral outward movement of control points ofcontrol objects. The method includes identifying a pair of startingpoints in respective centers of control points of one or more controlobjects that are detected in a 3D sensor space. In one implementation,the pair of starting points is a fixed distance apart. It also includesdetecting an outward expanding movement of the control points in the 3Dsensor space, identifying a pair of resting points in respective centersof the control points when the control points come to rest, defining ahorizontal extent of a virtual interaction space in proportion todistance between the starting points and the resting points, defining avertical extent of the virtual interaction space in proportion to widthof the control objects, and presenting the interaction space responsiveto the vertical extent and horizontal extent. In one implementation, thecontrol objects are hands and control points are finger tips.

In one implementation, a method is described for interacting with avirtual vector field in a 3D sensor space. The method includes defininga vector field at least responsive to curling of fingers of a hand anddegrees of freedom between fingers of the curled fingers. The vectorfield is centered with respect to a fixed point proximate to the handand magnitude of the vector field is calculated at least in part by ascale of curling of the fingers and degrees of freedom between thefingers. It also includes constructing a virtual sphere along aplurality of points on contour of curled fingers in the 3D sensor space,extending radially, inward or outward, one or more interaction vectorson the virtual sphere, wherein magnitudes of the interaction vectors aredetermined by radius of the virtual sphere, and compounding interactionsof the vector field with the interaction vector based on theirrespective magnitudes, wherein the interactions include at least one ofadding, multiplying, or taking dot-product of at least one vector in thevector field and the interaction vector.

In some implementations, a method is described for creating a virtualspring in a 3D sensor space. The method includes detecting a lateralmovement of a control object responsive to a lateral movement of a handin a 3D sensor space, defining a static length of a virtual spring thatis in proportion to length of the lateral movement, and defining aspring constant of the virtual spring at least responsive to curling offingers of the hand and degrees of freedom between fingers of the hand.The spring constant is centered with respect to a fixed point proximateto the curled fingers and magnitude of the spring constant is calculatedat least in part by a scale of curling of the fingers and degrees offreedom between the fingers. It further includes compoundinginteractions of the virtual spring with other virtual elements of the 3Dsensor space.

A method can be described for controlling a virtual camera in a 3Dsensor space. The method includes detecting a circular sweep around avirtual object responsive to a control gesture of a control object in a3D sensor space, calculating a radius of the circular sweep responsiveto a found point that is equidistant to a plurality of points defined oncontour of the control gesture, determining a focal length of a virtualcamera towards the virtual object responsive to the radius of thecircular sweep by constructing a virtual sphere in the 3D sensor spacethat is in proportion to the radius of the circular sweep, defining avector from the virtual camera to the center of the virtual sphere, anddetermining a point of intersection between the sphere and the vector.It also includes defining a field of view and orientation of the virtualcamera responsive to orientation of the control object and interpolatingthe virtual camera through time to a new position that coincides withthe point of intersection.

A method also can be described for spring-zooming a virtual camera in a3D sensor space. The method includes detecting a circular sweepresponsive to a first control gesture of a control object in a 3D sensorspace and calculating a radius of the circular sweep responsive to afound point that is equidistant to a plurality of points defined oncontour of the control gesture. The radius of the circular sweep definesa spring constant of a virtual camera launcher of a virtual camera and afirst distance between center of the circular sweep and the virtualcamera defines a static length of the spring movement. It also includesdetecting a backward pull of the virtual camera launcher to a seconddistance in response to a second control gesture of the control objectin the 3D space and accelerating the virtual camera through timeresponsive to releasing the virtual camera launcher by a third controlgesture. The control object is a hand and orientation of the virtualcamera is responsive to orientation of at least one finger of the hand.

A method can further be described for defining and controlling multiplevirtual cameras in a 3D sensor space. The method includes detectingcircular sweeps in response to control gestures of a control object in a3D sensor space, wherein the circular sweeps have respective centerpoints and direction vectors, constructing multiple virtual cameras inthe 3D space with different fields of view that are proportional torespective direction vectors of the circular sweeps, assigning each ofthe virtual cameras a virtual camera checkpoint from an array of virtualcamera selectors created in the 3D space by one or more controlgestures, and selecting and controlling visual parameters of aparticular virtual camera in response to selection of correspondingcamera selector. The visual parameters include at least position,orientation, focal length, deviation relative to the virtual camera, ormaximum aperture.

Some methods further include linking the virtual camera selectors to oneor more real camera in a physical space and selecting and controllingvisual parameters of a particular real camera in response to selectionof corresponding camera selector.

In one implementation, a method is described for manipulating a virtualcamera in a 3D sensor space. The method includes determining a focallength of a virtual camera in a 3D sensor space responsive to at leastone of radius of a circular sweep of hands, distance between midpointsof the hands, scale of curling of fingers of the hands, and degree offreedom between fingers. It also includes defining a field of view andorientation of the virtual camera responsive to orientation of thehands, constructing a virtual sphere along a plurality of points on anon-intersecting contour of the hands, defining a view vector from thecenter of the virtual sphere to a point on virtual sphere's surface thatis equidistant to a plurality of points on the hands, and manipulatingthe virtual camera by at least rotating, translating, compressing, orscaling the view vector responsive to subsequent control gestures of thehands.

In some implementations, a method is described for manipulating avirtual camera in a 3D sensor space. The method includes detecting afirst control gesture of a control object that defines a starting pointof a virtual camera in a 3D sensor space, detecting a second controlgesture of the control object that defines a continuous contour throughtime in the 3D sensor space, detecting a third control gesture of thecontrol object that defines a finishing point of the virtual camera inthe 3D sensor space, and moving the virtual camera along the continuouscontour between the starting point and the finishing point.

The method also can include determining a focal length of the virtualcamera responsive to distance of a finger of the hand from thecontinuous contour. The method also includes defining a field of viewand orientation of the virtual camera responsive to orientation of thefinger.

The method can further include mapping the continuous contour to astraight line and moving the virtual camera along the straight line. Themethod further includes defining a plurality of points on the continuouscontour to construct a Bezier curve responsive to respective sizes anddirections of the points.

In yet another implementation, a method is described for manipulatingvirtual objects in a 3D sensor space. The method includes creating avirtual vector field in response to a control gesture that makesswirling motions in a 3D sensor space, creating a plurality of virtualobjects in response to subsequent control gestures that make circularsweeps in the 3D sensor space and define object vectors on respectivevirtual objects, and compounding interactions of the vector field withthe object vectors based on their respective magnitudes, wherein theinteractions include at least one of adding, multiplying, or takingdot-product of at least one vector in the vector field and an objectvectors. In one implementation, the virtual vector field is a vortex. Inanother implementation, the size of the vortex is directly proportionalto scale of the swirling motions in the 3D space.

In some implementations, a method is described for performing augmentedinteractions with virtual objects in a 3D sensor space. The methodincludes creating a synthetic space by overlaying a virtual space on aphysical space, defining vectors on portions of the synthetic, virtual,and physical space, and compounding augmented interactions of vectors inthe physical space with vectors in the synthetic space, wherein theaugmented interactions modify at least one of positional, material, orother property of virtual objects in synthetic space. In oneimplementation, the interactions include at least one of adding,multiplying, or taking dot-product of at least one vector in thephysical space and at least one vector in the synthetic space.

Among other aspects, implementations can enable improved control ofmachines or other computing resources based at least in part upondetermining whether positions and/or motions of an object (e.g., hand,tool, hand and tool combinations, other detectable objects orcombinations thereof) might be interpreted as an interaction with one ormore virtual objects. Implementations can enable modeling of physicalobjects, created objects and interactions with combinations thereof forinterfacing with a variety of machines (e.g., a computing systems,including desktop, laptop, tablet computing devices, special purposecomputing machinery, including graphics processors, embeddedmicrocontrollers, gaming consoles, audio mixers, or the like; wired orwirelessly coupled networks of one or more of the foregoing, and/orcombinations thereof).

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the subject matter can be derived byreferring to the detailed description and claims when considered inconjunction with the following Figures, wherein like reference numbersrefer to similar elements throughout the Figures.

FIG. 1 illustrates a three-dimensional multi-stroke user-definedinteraction widget.

FIG. 2 illustrates an interaction space whose size is defined inproportion to a radius of a user's stroke.

FIG. 3 shows an interaction space that is defined in response to outwardexpanding movement of hands.

FIG. 4 illustrates a vector field defined by the curvature of a user'shand.

FIG. 4B illustrates potential gravitational attractors in accordancewith an implementation.

FIG. 5 illustrates a user-defined spring interaction element in a threedimensional (3D) sensor space.

FIGS. 6A, 6B, 6C, 6D, and 6E show one implementation of controlling avirtual camera in a three dimensional (3D) sensor space.

FIGS. 7A, 7B, 7C, 8A, 8B, 8C, 8D, and 8E illustrate an example machinesensory and control system according to implementations.

FIG. 9 illustrates a sensory augmentation system to add simulatedsensory information to a virtual reality input according to animplementation.

FIG. 10 illustrates an example computing system according to animplementation.

FIGS. 11, 11A, 11B, and 11C illustrate one implementation of a springzooming camera movement of a virtual camera in a three dimensional (3D)sensor space.

FIGS. 12A, 12B, and 12C illustrate defining and controlling multiplevirtual cameras in a three dimensional (3D) sensor space.

FIG. 13 illustrates pluck and release camera controls in a threedimensional (3D) sensor space.

FIG. 14 illustrates a sphere grabbing camera manipulation in a threedimensional (3D) sensor space.

FIGS. 15A, 15B, and 15C illustrate path creation camera controls in athree dimensional (3D) sensor space.

FIG. 16 illustrates an implementation where vectors on a control portionof a user's hand interact with vectors in the virtual field.

FIG. 17 illustrates an augmented reality application where a virtualspace is overlaid on a physical space to create a synthetic space.

FIG. 18 illustrates an augmented interaction that is used to navigate amenu system.

DETAILED DESCRIPTION

Techniques described herein can be implemented as one or a combinationof methods, systems or processor executed code to form implementationscapable of improved control of machines or other computing resourcesbased at least in part upon determining whether positions and/or motionsof an object (e.g., hand, tool, hand and tool combinations, otherdetectable objects or combinations thereof) might be interpreted as aninteraction with one or more virtual objects. Implementations can enablemodeling of physical objects, created objects and interactions withcombinations thereof for machine control or other purposes.

A user can interact with a device incorporating a 3D sensor such asdescribed in U.S. Prov. App. No. 61/816,487 and U.S. Prov. App. No.61/872,538 by using gestures in a 3D sensor space monitored by the 3Dsensor. Interacting with the device often requires the control object(e.g., a hand) exiting the 3D sensor space (a “resetting” gesture) tospecify a control (or engagement of a control) of the device. Thetechnology disclosed relates to methods for interpreting gestures of acontrol object in a 3D sensor space, without requiring the controlobject exiting the 3D sensor space.

In this application, a 3D interaction space is part of a sensor space. Asensor space is a 3D volume in which a sensor, such as an upward lookingbinocular sensor, can track gestures of a control object. One controlobject can be a hand, including the palm, fingers and thumb. Anothercontrol object can be a pointer.

Gesture tracking involves tracking multiple dimensions of gestures madewith the control object. The overall path of the control object throughthree-dimensional space is tracked. The speed and acceleration withwhich the control object moves is tracked. When the control object is ahand or other object with appendages, multiple degrees of freedom fororientation of the hand and of the individual fingers are tracked.

Gesture tracking can involve measuring additional parameters of thegesture. The sections that follow identify parameters of variousgestures. Examples of gesture parameters for a control object such as ahand that can be characterized include a twist of the wrist, anorientation of the hand relative to the control surface, an orientationof the palm or back of the hand, positions of fingers relative to thepalm, and positions of fingers relative to one another. In this sense, athumb can be considered one of the fingers or an opposable thumb mayhave a special meaning distinct from the meaning of fingers. Individualfingers can have individual meanings.

Gestures link to controls or content that can be visualized with avisual display. In some implementations, the visual display begins withcontrols that become connected to gestures. For instance, a graphic userinterface that has controls can be connected to gestures in theinteraction space that manipulate the controls. In otherimplementations, gestures cause controls to appear on the visual displayand then allow the user to interact with those new controls. Forinstance, applying gestures and interaction spaces to augmented realitycan involve users superimposing controls or content over real scenescreating controls or display areas in thin air. The position of thesuperimposed controls can remain constant as the viewer looks around.Similarly, augmented virtual reality may involve users superimposingcontrols over virtual scenes.

Gestures take on meaning in context. In some implementations, context isset before the gesture is made. Context can be selected with keystrokes,spoken commands, eye movement, facial expressions, gestures of controlobjects and the like. In other implementations, context is inferred fromthe gesture. When a series of gestures are made, some of the gesturescan be dedicated to setting a context for subsequent gestures.

Gestures can also link to virtual cameras in the sensor space. A virtualcamera's properties such as focal length, position, orientation, ormovement can be connected to the gestures. In one instance, curling ofthe fingers of a hand can be used to define zoom level of a virtualcamera in the sensor space.

This general framework can be, but is not necessary, to the variousgesture implementations described below.

Stroke

FIG. 1 illustrates a three-dimensional multi-stroke user-definedinteraction widget. A stroke is represented by a collection of pointsoutput by a Machine Sensory and Control System (MSCS) FIGS. 7A, 7B, 7C,8A, 8D, and 8E. This collection of points has at least one parameter incommon within some threshold deviation. In one implementation, a strokecan be defined by a group of points which are continuous in time withvelocity vectors within a threshold angle of each other. In anotherimplementation, such velocity vectors can represent the differencevector between successive positions of the control object at points intime. In yet another implementation, a stroke can include a collectionof points within a given radial distance to a fixed point.

FIG. 1 shows an implementation where the height 180 of an interactionspace is defined in proportion to the height of a stroke sensed in aninteraction space defining context. In some implementations, any spatialattribute of an interaction space can be defined in proportion to anydefining characteristic of a stroke. At least one parameter of a secondstroke can defines a further attribute of the interaction space, such asits width 182, so that the first stroke defines the height and thesecond stroke the width of an interaction space. At least one parameterof a third stroke can define a further attribute of the interactionspace, such as its depth 181. Interaction elements (e.g. but withoutlimitation buttons, dials, panels etc.) placed within such auser-defined interaction space can expand or shrink in a predeterminedway in proportion to the dimensions of the interaction space.

Density of a control object can refer to concentration of skin colorpigments on a user appendage such as a hand. In another implementation,scale of the control object refers to level of relative separation ofcontrol points in a control object like fingertips in a hand.

Interaction includes a location in virtual space; in implementationsthis virtual space may be associated with a physical space for exampleas described in commonly owned U.S. Provisional Patent ApplicationsAttorney Docket No. 1008-1, entitled “Velocity Field Interaction forFree Space Gesture Interface and Control” to Isaac Cohen (61/891,880).An interaction can include one or more quantities representing variousattributes, such as for example a manipulation point “strength”attribute.

Definition of an interaction space can either be linked to an existingcontrol or can cause a control to appear. For instance, a pull down menucan be linked to an interaction space. A small interaction space can bedefined for just a few items. A large interaction space can make iteasier to select from a long list of items. In augmented reality, aninteraction space can cause a window to be superimposed over a realscene. The window can be filled with predetermined content or a user canselect the content to be displayed in the window. The window also oralternatively can be filled with controls.

Radial Gesture

FIG. 2 illustrates in frame 1 a creation gesture to create aninteraction space. Now with reference to frame 2, an interaction spaceis specified to include one or more elements (saw tooth lines). Now withreference to frame 3, an interaction space whose size is defined inproportion to a radius of a user's stroke 230 in a creation gesture. Inone implementation, radius can refer to a radius about a found pointthat is equidistant to a plurality of points defined on the strokecontour 228. In some implementations, as shown in frames 3 and 4, aradial dial can be defined in relation to the stroke contour andsubsequently manipulated. In other implementations, the radial gesturedefines a circular window that is superimposed over a real scene andpopulated with content or controls.

Hand Separation

Implementations can permit the use of two-handed manipulations ofvirtual objects. A user can hold a virtual object in place with one handwhile manipulating the object with the other hand. Users can stretch,shrink, contort and otherwise transform virtual objects in the same waysas the virtual object manipulations. A virtual construct (i.e., plane)can be defined in proximity to the virtual object to enable engagementswith the object. One use of such virtual constructs is further describedin commonly owned U.S. Provisional Patent Applications Nos. 61/825,480,61/825,418, 61/873,351, 61/877,641, 61/825,515. Real and/or virtualobjects can be used in conjunction with a manipulated object. Forexample a real or virtual keyboard can be used with a virtual screen.

FIG. 3 illustrates an implementation where two user appendages 310, 312,as identified by an MSCS, start at a fixed distance apart, expandoutward, and then come to rest. The length at which they come to restdefines at least one parameter of a user interaction space. In oneimplementation, points defined at the center of the users palms are usedto measure the outward expanding gesture. In another implementation, thetwo appendages are two fingertips of one hand.

Thus for example as illustrated in FIG. 3, a creation gesture 320comprising drawing apart of hands 310, 312 defines one dimension of aninteraction space 330, in this example the horizontal or (“length”)dimension. Other dimensions can be specified gesturally using similarmotions of hands 310, 312 to specify different dimensions of theinteraction space.

Yet further, in FIG. 3, a creation gesture 328 comprising a pointing onby a single finger of hand 310 creates a new virtual object 340 in theinteraction space 330. In implementations, creation of virtual objectscan include describing a perimeter or circumference using a pointingfinger, or fist or the like. In other implementations, virtual objectsof a particular set size can be created by a tapping of a finger orfist.

Gravitational Attractor

FIG. 4A illustrates a vector field defined at least in part by thecurvature of a user's hand 476 and centered with respect to some pointproximate to the hand. In one implementation, the vector is defined inresponse to curling of fingers of a hand and degrees of freedom orseparation between fingers of the curled fingers. In anotherimplementation, a sphere of best-fit is fit to a plurality of points ona user's hand as detected by the MSCS. The radius of this sphere 474 isused, along with other optional parameters, to define the magnitude of avector on the sphere. In some implementations, at least one vectorextends radially outward or inward between the center and surface of thesphere. In some applications, user-hand defined vector fields can enablecompound interactions with the vector fields in interaction spaces isfurther described in commonly owned U.S. patent application No.14/516,493, filed 16 Oct. 2014 (Attorney Docket No. LEAP1008-2/LPM-1008US). In one implementation, such interactions includeadding, multiplying, or taking the dot-product of at least one vector onthe sphere and at least one vector in an interaction space. These vectoroperations can be applied to tensors represented by vectors.

Tensors can be a kind of vector and tensor spaces can be implementationsof vector spaces. A tensor can include stress, strain, shear, or otherobject properties which can describe complex interactions with virtualobjects. In one implementation, the vectors can include tensors. Suchtensors can describe material properties of object portions in thevirtual, physical, synthetic space, or any combination, such as stress,strain, shear, or other material properties.

Further, a vector field can be based upon virtual forces (e.g., virtualgravity, virtual electromagnetism, virtual charisma, etc.) enablinginteractions with virtual objects over distances. For example, a“gravity grab” interaction in an astronomy genre gaming engine orphysics teaching implementations includes emulating the force of gravityby selecting a function in which the strength is proportional to a“virtual mass” of the virtual object but declines with the square of thedistance between the hand and the virtual object. In implementationsemploying strength to emulate virtual properties of objects, virtualflexibility/rigidity enable interactions with virtual objects emulatingone type of material to have different interactions than virtual objectsemulating another type of material. For example, a virtual steel spherewill behave differently to a virtual “squeeze” than a virtual rubbersphere. Virtual properties (e.g., virtual mass, virtual distance,virtual flexibility/rigidity, etc.) and virtual forces (e.g., virtualgravity, virtual electromagnetism, virtual charisma, etc.), like virtualobjects, can be created (i.e., having no analog in the physical world)or modeled (i.e., having an analog in the physical world). Normalvectors or gradients can be used.

FIG. 4B illustrates potential gravitational attractors in accordancewith an implementation. In frame 1 of FIG. 4B is depicted animplementation in which depending on the direction of the creationgesture, the sign of the force from the gravitation may be reversed. Forexample, clockwise 474 and counter clockwise 476 spiral gestures can beused to specify for example a positive (e.g., inward) gravitation G1 478and a negative (e.g., outward) gravitation G2 480, respectively. Inframe 2, another possible implementation places a gravitationalattractor G 484 at the tip of the active finger 486. In frame 3, animplementation is depicted wherein when the camera 492 is moved, theuser may also move their finger 494 to alter the force on the camera.

Spring Interaction

FIG. 5 illustrates a user-defined virtual system that reflectsproperties of physical systems. In the example of FIG. 5, a springinteraction element is defined virtually in a three dimensional (3D)sensor space. In one implementation, one stroke 510 is used to definethe length of the spring 516. In other implementations, one stroke 530,the opening of the fingers from the thumb, is used to define at leastone other parameter of the spring, e.g., a stiffness of the spring 536which, in this example, is designated as ‘k’. Once-defined, such springcan be attached to and interact with other user-interaction elements ofthe 3D space.

Circle Tween Camera Movement

FIGS. 6A-6E show one implementation of controlling a virtual camera in athree dimensional (3D) sensor space. FIG. 6A illustrates a “tween”function for camera movement in 4 Frames. In Frame 1, an object ofinterest 606 is disposed in view of camera 608. In Frame 2 a userdefines a stroke 616 relative to the movement of at least one parameterof a hand through time, as captured by a MSCS, around the object ofinterest 606 identified in Frame 1. In one implementation, a userdescribes any non-intersecting contour around an object of interest. Inanother implementation, a circle of best-fit is fit to a plurality ofpoints on this contour. Frame 3 illustrates the placement of a camera622 on the edge of the sphere defined in Frame 2. Frame 4 illustrateshow the radius of the circle 616 can be used to control the precessionof the camera 622 around the object of interest 606.

FIG. 6B illustrates how the radius of a stroke determines the “zoom”level and orientation of a camera control in 4 Frames. A circle is drawnaround an object of interest 640 in Frame 1. In one implementation, auser describes any non-intersecting contour around an object ofinterest. In another implementation, a circle of best-fit is fit to aplurality of points on this contour. In Frame 2, from the circle anequator is determined for a sphere 644, which encloses the object ofinterest 640. In a Frame 3, a vector 648 is defined from the camera'scurrent position to the center of the enclosing sphere. In a Frame 4, apoint I on the circle with radius r 642 can then be calculated bysolving the integration point of vector a 648 and the sphere equation.

Now with reference to FIG. 6C, shown is the “tween” function in frames5, 6, which illustrate how a camera's position is interpolated throughtime as it moves “tween” two points, beginning with a fore point 664 andeventually reaches the intersection point I 662 on the sphere's surface.In other applications, the camera instantly moves from one location to anew location. As shown in Frame 6, depending on the size of the circledrawn, the radius of the sphere 672 and the fore point 664, theintersection point ‘I’ 662 will be further away or closer to the object640, which allows a definition of camera position relative to the objectas closely as the user desires to specify it.

FIGS. 6D-6E show another example of controlling a virtual camera in athree dimensional space. Applying the technology disclosed, more thanone hand attribute (velocity vector, palm normal, curvature of a finger,rotation matrix of hand, etc., and combinations) in the 3D sensor spacecan be mapped to more than one context hierarchy at the same time by thecomputing device with the 3D sensor. For example, frames 1-3 illustratemoving a camera from position P 676 to position O 677 by “stroking” avector field with the palm as if pushing water. As shown in frame 1, theuser positions their palm normal vector n approximately matching thepalm velocity vector v, and strokes the field as if pushing water sothat the object O moves closer to point P. In one implementation, pointP is actually moved, however, the result is the same whether point P orobject O is moved. In frame 2, the user's palm normal vector n ispositioned approximately perpendicular to the palm velocity vector v,when the user simply slides their hand back through the field, makingsure the palm normal does not match the direction of the palm velocityin order to reset the position of the hand. In frame 3, it is shown thatthe combination of the movement types in frames 1 and 2 provide anoverall trajectory (e.g., the palm engages the field when gesture inputis desired, disengages from the field when returning to a startingposition within the view of the camera.

FIG. 6E illustrates the application of the technology disclosed, wherethe computing device automatically interprets a gesture of a controlobject 687 in a 3D sensor space by discerning a control plane 686 of thecontrol object 687, according to one implementation. The computingdevice first senses a control object such as a user's hand 687 in the 3Dsensor space. The computing device then senses an orientation of thecontrol object 687 and determines a surface of the control object 687.For example, a surface of a hand 687 can be the palm back of the hand687. The computing device defines a palm normal plane 685 that has anorientation to the surface of the control object 687.

The computing device then interprets a gesture in the 3D sensor spacebased on whether the movement of the palm normal plane 685 is morenormal to the control plane 686 or more parallel to the control plane686. In some implementations, the computing device calculates atrajectory (an angular trajectory) of the movement of the palm normalplane, and determines whether the gesture engages a virtual controlbased on whether the trajectory is more normal or more parallel to thecontrol plane 686.

FIG. 6D illustrates in frame 1 that the palm normal plane 679 is morenormal to the control object's trajectory 678. The palm normal plane 679is more normal to the trajectory 687 of FIG. 6E when a normal vector ofthe palm normal plane 679 is within a pre-determined range from atangent vector of the trajectory 678 intersecting the palm normal plane679. For example, the control plane 679 is more normal to the trajectory678 when the normal vector of the palm normal plane 679 is within +/−10degrees from the tangent vector of the trajectory 678. For example, thepalm normal plane 679 is more normal to the trajectory 678 when thenormal vector of the palm normal plane 679 is within +/−20 degrees orwithin +/−30 degrees from the tangent vector of the trajectory 678.

FIG. 6D depicts in frame 2 that the palm normal plane 681 is moreperpendicular to the control object's trajectory 680. The palm normalplane 681 is more perpendicular to the trajectory 680 when the palmnormal plane 681 is within a pre-determined range from a perpendicularvector of the trajectory 680 intersecting the palm normal plane 681. Inone example, the palm normal plane 681 is more perpendicular to thetrajectory 680 when the palm normal plane 681 is within +/−10 degreesfrom the perpendicular vector of the trajectory 680. In another example,the palm normal plane 681 is more perpendicular to the trajectory 680when the palm normal plane 681 is within +/−20 degrees or within +/−30degrees from the perpendicular vector of the trajectory 680.

Moving the control object 687 back through the three dimensional spaceso that the velocity of the hand does not match the normal of the palmresets the position of the hand. Additional examples include traversingmenus with one hand and traversing menu paths with more than one hand.For example, a user can use one hand to change channel and the otherhand to set volume at the same time. Another example has a user canchanging channel by pushing with one hand, while turning down the volumeby rotation motion of a finger on the one hand.

Machine Sensory and Control System

FIGS. 7A, 7B, 7C, 8A, 8D, and 8E illustrate an example machine sensoryand control system in embodiments. In one embodiment, a motion sensingand controller system provides for detecting that some variation(s) inone or more portions of interest of a user has occurred, for determiningthat an interaction with one or more machines corresponds to thevariation(s), for determining if the interaction should occur, and, ifso, for affecting the interaction. The Machine Sensory and ControlSystem (MSCS) typically includes a portion detection system, a variationdetermination system, an interaction system and an application controlsystem.

As FIG. 7A shows, one detection system 90A embodiment includes anemission module 91, a detection module 92, a controller 96, a processingmodule 94 and a machine control module 95. In one embodiment, theemission module includes one or more emitter(s) 181A, 181B (e.g., LEDsor other devices emitting light in the IR, visible, or other spectrumregions, or combinations thereof; radio and/or other electromagneticsignal emitting devices) that are controllable via emitter parameters(e.g., frequency, activation state, firing sequences and/or patterns,etc.) by the controller 96. However, other existing/emerging emissionmechanisms and/or some combination thereof can also be utilized inaccordance with the requirements of a particular implementation. Theemitters 180A, 180B can be individual elements coupled with materials ordevices 182 (and/or materials) (e.g., lenses 182, multi-lenses (of FIG.7A), image directing film (IDF) 182C (of FIG. 7B), liquid lenses,combinations thereof, and/or others) with varying or variable opticalproperties to direct the emission, one or more arrays 180C of emissiveelements (combined on a die or otherwise), with or without the additionof devices 182C for directing the emission, or combinations thereof, andpositioned within an emission region 181 (of FIG. 7B) according to oneor more emitter parameters (i.e., either statically (e.g., fixed,parallel, orthogonal or forming other angles with a work surface, oneanother or a display or other presentation mechanism) or dynamically(e.g., pivot, rotate and/or translate) mounted, embedded (e.g., within amachine or machinery under control) or otherwise coupleable using aninterface (e.g., wired or wireless)). In some embodiments, structuredlighting techniques can provide improved surface feature capturecapability by casting illumination according to a reference pattern ontothe object 98. Image capture techniques described in further detailherein can be applied to capture and analyze differences in thereference pattern and the pattern as reflected by the object 98. In yetfurther embodiments, detection system 90 A may omit emission module 91altogether (e.g., in favor of ambient lighting).

In one embodiment, the detection module 92 includes one or more capturedevice(s) 190A, 190B (e.g., light (or other electromagnetic radiationsensitive devices) that are controllable via the controller 96. Thecapture device(s) 190A, 190B can comprise individual or multiple arraysof image capture elements 190A (e.g., pixel arrays, CMOS or CCD photosensor arrays, or other imaging arrays) or individual or arrays ofphotosensitive elements 190B (e.g., photodiodes, photo sensors, singledetector arrays, multi-detector arrays, or other configurations of photosensitive elements) or combinations thereof. Arrays of image capturedevice(s) 190C (of FIG. 7C) can be interleaved by row (or column or apattern or otherwise addressable singly or in groups). However, otherexisting/emerging detection mechanisms and/or some combination thereofcan also be utilized in accordance with the requirements of a particularimplementation. Capture device(s) 190A, 190B each can include aparticular vantage point 190-1 from which objects 98 within area ofinterest 5 are sensed and can be positioned within a detection region191 (of FIG. 7C) according to one or more detector parameters (i.e.,either statically (e.g., fixed, parallel, orthogonal or forming otherangles with a work surface, one another or a display or otherpresentation mechanism) or dynamically (e.g. pivot, rotate and/ortranslate), mounted, embedded (e.g., within a machine or machinery undercontrol) or otherwise coupleable using an interface (e.g., wired orwireless)). Capture devices 190A, 190B can be coupled with devices 192(and/or materials) (of FIG. 7C) (e.g., lenses 192A (of FIG. 7C),multi-lenses 192B (of FIG. 7C), image directing film (IDF) 192C (of FIG.7C), liquid lenses, combinations thereof, and/or others) with varying orvariable optical properties for directing the reflectance to the capturedevice for controlling or adjusting resolution, sensitivity and/orcontrast. Capture devices 190A, 190B can be designed or adapted tooperate in the IR, visible, or other spectrum regions, or combinationsthereof; or alternatively operable in conjunction with radio and/orother electromagnetic signal emitting devices in various applications.In an embodiment, capture devices 190A, 190B can capture one or moreimages for sensing objects 98 and capturing information about the object(e.g., position, motion, etc.). In embodiments comprising more than onecapture device, particular vantage points of capture devices 190A, 190Bcan be directed to area of interest 5 so that fields of view 190-2 ofthe capture devices at least partially overlap. Overlap in the fields ofview 190-2 provides capability to employ stereoscopic vision techniques(see, e.g., FIG. 7-2), including those known in the art to obtaininformation from a plurality of images captured substantiallycontemporaneously.

While illustrated with reference to a particular embodiment in whichcontrol of emission module 91 and detection module 92 are co-locatedwithin a common controller 96, it should be understood that thesefunctions will be separate in some embodiments, and/or incorporated intoone or a plurality of elements comprising emission module 91 and/ordetection module 92 in some embodiments. Controller 96 comprises controllogic (hardware, software or combinations thereof) to conduct selectiveactivation/de-activation of emitter(s) 180A, 180B (and/or control ofactive directing devices) in on-off, or other activation states orcombinations thereof to produce emissions of varying intensities inaccordance with a scan pattern which can be directed to scan an area ofinterest 5. Controller 96 can comprise control logic (hardware, softwareor combinations thereof) to conduct selection, activation and control ofcapture device(s) 190A, 190B (and/or control of active directingdevices) to capture images or otherwise sense differences in reflectanceor other illumination. Signal processing module 94 determines whethercaptured images and/or sensed differences in reflectance and/or othersensor-perceptible phenomena indicate a possible presence of one or moreobjects of interest 98, including control objects 99, the presenceand/or variations thereof can be used to control machines and/or otherapplications 95.

In various embodiments, the variation of one or more portions ofinterest of a user can correspond to a variation of one or moreattributes (position, motion, appearance, surface patterns) of a userhand 99, finger(s), points of interest on the hand 99, facial portion 98other control objects (e.g., styli, tools) and so on (or somecombination thereof) that is detectable by, or directed at, butotherwise occurs independently of the operation of the machine sensoryand control system. Thus, for example, the system is configurable to‘observe’ ordinary user locomotion (e.g., motion, translation,expression, flexing, deformation, and so on), locomotion directed atcontrolling one or more machines (e.g., gesturing, intentionallysystem-directed facial contortion, etc.), attributes thereof (e.g.,rigidity, deformation, fingerprints, veins, pulse rates and/or otherbiometric parameters). In one embodiment, the system provides fordetecting that some variation(s) in one or more portions of interest(e.g., fingers, fingertips, or other control surface portions) of a userhas occurred, for determining that an interaction with one or moremachines corresponds to the variation(s), for determining if theinteraction should occur, and, if so, for at least one of initiating,conducting, continuing, discontinuing and/or modifying the interactionand/or a corresponding interaction.

For example and with reference to FIG. 8A, a variation determinationsystem 90B embodiment comprises a model management module 197 thatprovides functionality to build, modify, customize one or more models torecognize variations in objects, positions, motions and attribute stateand/or change in attribute state (of one or more attributes) fromsensory information obtained from detection system 90A. A motion captureand sensory analyzer 197E finds motions (i.e., translational,rotational), conformations, and presence of objects within sensoryinformation provided by detection system 90A. The findings of motioncapture and sensory analyzer 197E serve as input of sensed (e.g.,observed) information from the environment with which model refiner 197Fcan update predictive information (e.g., models, model portions, modelattributes, etc.).

A model management module 197 embodiment comprises a model refiner 197Fto update one or more models 197B (or portions thereof) from sensoryinformation (e.g., images, scans, other sensory-perceptible phenomenon)and environmental information (i.e., context, noise, etc.); enabling amodel analyzer 197I to recognize object, position, motion and attributeinformation that might be useful in controlling a machine. Model refiner197F employs an object library 197A to manage objects including one ormore models 197B (i.e., of user portions (e.g., hand, face), othercontrol objects (e.g., styli, tools)) or the like (see e.g., model197B-1, 197B-2 of FIGS. 8B, 8C)), model components (i.e., shapes, 2Dmodel portions that sum to 3D, outlines 194 and/or outline portions194A, 194B (i.e., closed curves), attributes 197-5 (e.g., attach points,neighbors, sizes (e.g., length, width, depth), rigidity/flexibility,torsional rotation, degrees of freedom of motion and others) and soforth) (see e.g., 197B-1-197B-2 of FIGS. 8B 8C), useful to define andupdate models 197B, and model attributes 197-5. While illustrated withreference to a particular embodiment in which models, model componentsand attributes are co-located within a common object library 197A, itshould be understood that these objects will be maintained separately insome embodiments.

FIG. 8B illustrates prediction information including a model 197B-1 of acontrol object (e.g., FIG. 7A: 99) constructed from one or more modelsubcomponents 197-2, 197-3 selected and/or configured to represent atleast a portion of a surface of control object 99, a virtual surfaceportion 194 and one or more attributes 197-5. Other components can beincluded in prediction information 197B-1 not shown in FIG. 8B forclarity sake. In an embodiment, the model subcomponents 197-2, 197-3 canbe selected from a set of radial solids, which can reflect at least aportion of a control object 99 in terms of one or more of structure,motion characteristics, conformational characteristics, other types ofcharacteristics of control object 99, and/or combinations thereof. Inone embodiment, radial solids include a contour and a surface defined bya set of points having a fixed distance from the closest correspondingpoint on the contour. Another radial solid embodiment includes a set ofpoints normal to points on a contour and a fixed distance therefrom. Inan embodiment, computational technique(s) for defining the radial solidinclude finding a closest point on the contour and the arbitrary point,then projecting outward the length of the radius of the solid. In anembodiment, such projection can be a vector normal to the contour at theclosest point. An example radial solid (e.g., 197-3) includes a“capsuloid”, i.e., a capsule shaped solid including a cylindrical bodyand semi-spherical ends. Another type of radial solid (e.g., 197-2)includes a sphere. Other types of radial solids can be identified basedon the foregoing teachings.

One or more attributes 197-5 can define characteristics of a modelsubcomponent 197-3. Attributes can include e.g., attach points,neighbors, sizes (e.g., length, width, depth), rigidity, flexibility,torsion, zero or more degrees of freedom of motion with respect to oneor more defined points, which can include endpoints for example, andother attributes defining a salient characteristic or property of aportion of control object 99 being modeled by predictive information197B-1. In an embodiment, predictive information about the controlobject can include a model of the control object together withattributes defining the model and values of those attributes.

In an embodiment, observation information including observation of thecontrol object can be compared against the model at one or more ofperiodically, randomly or substantially continuously (i.e., in realtime). Observational information can include without limitation observedvalues of attributes of the control object corresponding to theattributes of one or more model subcomponents in the predictiveinformation for the control object. In an embodiment, comparison of themodel with the observation information provides an error indication. Inan embodiment, an error indication can be computed by determining aclosest distance determined between a first point A belonging to a setof points defining the virtual surface 194 and a second point Bbelonging to a model subcomponent 197-2 determined to be correspondingto the first point (e.g., nearest to the first point for example). In anembodiment, the error indication can be applied to the predictiveinformation to correct the model to more closely conform to theobservation information. In an embodiment, error indication can beapplied to the predictive information repeatedly until the errorindication falls below a threshold, a measure of conformance with theobservation information rises above a threshold, or a fixed or variablenumber of times, or a fixed or variable number of times per time period,or combinations thereof.

In an embodiment and with reference to FIGS. 7A, 8C, updating predictiveinformation to observed information comprises selecting one or more setsof points (e.g., FIG. 8C: 193A, 193B) in space surrounding or boundingthe control object within a field of view of one or more image capturedevice(s). As shown by FIG. 8C, points 193 can be determined using oneor more sets of lines 195A, 195B, 195C, and 195D originating at vantagepoint(s) (e.g., FIG. 7A: 190-1, 190-2) associated with the image capturedevice(s) (e.g., FIG. 7A: 190A-1, 190A-2) and determining therefrom oneor more intersection point(s) defining a bounding region (i.e., regionformed by lines FIG. 8C: 195A, 195B, 195C, and 195D) surrounding across-section of the control object. The bounding region can be used todefine a virtual surface (FIG. 8C: 194) to which model subcomponents197-1, 197-2, 197-3, and 197-4 can be compared. The virtual surface 194can include a visible portion 194A and a non-visible “inferred” portion194B. Virtual surfaces 194 can include straight portions and/or curvedsurface portions of one or more virtual solids (i.e., model portions)determined by model refiner 197F on FIG. 8A.

For example and according to one embodiment illustrated by FIG. 8C,model refiner 197F determines to model subcomponent 197-1 of an objectportion (happens to be a finger) using a virtual solid, an ellipse inthis illustration, or any of a variety of 3D shapes (e.g., ellipsoid,sphere, or custom shape) and/or 2D slice(s) that are added together toform a 3D volume. Accordingly, beginning with generalized equations foran ellipse (1) with (x, y) being the coordinates of a point on theellipse, (x_(C), y_(C)) the center, a and b the axes, and θ the rotationangle. The coefficients C₁, C₂ and C₃ are defined in terms of theseparameters, as shown:

$\begin{matrix}{{{{C_{1}x^{2}} + {C_{2}{xy}} + {C_{3}{y^{2} \cdot \left( {{2C_{1}x_{c}} + {C_{2}y_{c}}} \right)}{x \cdot \left( {{2C_{3}y_{c}} + {C_{2}x_{c}}} \right)}y} + \left( {{C_{1}x_{c}^{2}} + {C_{2}x_{c}y_{c}} + {C_{3}y_{c}^{2}} - 1} \right)} = 0}\mspace{20mu} {C_{1} = {\frac{\cos^{2}\theta}{a^{2}} + \frac{\sin^{2}\theta}{b^{2}}}}\mspace{20mu} {C_{2} = {{- 2}\cos \; \theta \; \sin \; {\theta \left( {\frac{1}{a^{2}} - \frac{1}{b^{2}}} \right)}}}\mspace{20mu} {C_{3} = {\frac{\sin^{2}\theta}{a^{2}} + \frac{\cos^{2}\theta}{b^{2\;}}}}} & (1)\end{matrix}$

The ellipse equation (1) is solved for θ subject to the constraintsthat: (1) (x_(C), y_(C)) must lie on the centerline determined from thefour tangents 195A, 195B, 195C, and 195D (i.e., centerline 189A of FIG.8C); and (2) is fixed at the assumed value a₀. The ellipse equation caneither be solved for θ analytically or solved using an iterativenumerical solver (e.g., a Newtonian solver as is known in the art). Ananalytic solution can be obtained by writing an equation for thedistances to the four tangent lines given a y_(C) position, then solvingfor the value of y_(C) that corresponds to the desired radius parametera=a₀. Accordingly, equations (2) for four tangent lines in the x-y plane(of the slice), in which coefficients A_(i), B_(i) and D_(i) (for i=1 to4) are determined from the tangent lines 195A, 195B, 195C, and 195Didentified in an image slice as described above.

A ₁ ×+B ₁ y+D ₁₌0A ₂ ×+B ₂ y+D ₂₌0A ₃ ×+B ₃ y+D ₃₌0A ₄ ×+B ₄ y+D₄₌₀  (2)

Four column vectors r₁₂, r₂₃, r₁₄ and r₂₄ are obtained from thecoefficients A_(i), B_(i) and D_(i) of equations (2) according toequations (3), in which the “\” operator denotes matrix left division,which is defined for a square matrix M and a column vector v such thatM\v=r, where r is the column vector that satisfies Mr=v:

$\begin{matrix}{r_{13} = {\begin{bmatrix}A_{1} & B_{1} \\A_{3} & B_{3}\end{bmatrix}\left. {\backslash\lbrack}\begin{matrix}{- D_{1}} \\{- D_{3\;}}\end{matrix} \right\rbrack}} & (3) \\{r_{23} = {\begin{bmatrix}A_{2} & B_{2} \\A_{3} & B_{3}\end{bmatrix}\left. {\backslash\lbrack}\begin{matrix}{- D_{21}} \\{- D_{3}}\end{matrix} \right\rbrack}} & \; \\{r_{14} = {\begin{bmatrix}A_{1} & B_{1} \\A_{4} & B_{4}\end{bmatrix}\left. {\backslash\lbrack}\begin{matrix}{- D_{1}} \\{- D_{4}}\end{matrix} \right\rbrack}} & \; \\{r_{24} = {\begin{bmatrix}A_{2} & B_{2} \\A_{4} & B_{4}\end{bmatrix}\left. {\backslash\lbrack}\begin{matrix}{- D_{2}} \\{- D_{4}}\end{matrix} \right\rbrack}} & \;\end{matrix}$

Four component vectors G and H are defined in equations (4) from thevectors of tangent coefficients A, B and D and scalar quantities p andq, which are defined using the column vectors r₁₂, r₂₃, r₁₄ and r₂₄ fromequations (3).

c1=(r ₁₃ +r ₂₄)/2c2=(r ₁₄ +r ₂₃)/2δ1=c2₁ −c1₁ δ2=c2₂ −c1₂ p=δ1/δ2p=q=c1₁−c1₂ *p G=Ap+B H=Aq+D  (4)

Six scalar quantities v_(A2), v_(AB), v_(B2), w_(A2), w_(AB), and w_(B2)are defined by equation (5) in terms of the components of vectors G andH of equation (4).

$\begin{matrix}{{v = {\begin{bmatrix}G_{2}^{2} & G_{3}^{2} & G_{4}^{2} \\\left( {G_{2}H_{2}} \right)^{2} & \left( {G_{3}H_{3}} \right)^{2} & \left( {G_{4}\; H_{4}} \right)^{2} \\H_{2}^{2} & H_{3}^{2} & H_{4\;}^{2}\end{bmatrix}\left. {\backslash\lbrack}\begin{matrix}0 \\0 \\1\end{matrix} \right\rbrack}}{w = {\begin{bmatrix}G_{2}^{2} & G_{3}^{2} & G_{4}^{2} \\\left( {G_{2}H_{2}} \right)^{2} & \left( {G_{3}H_{3}} \right)^{2} & \left( {G_{4}H_{4}} \right)^{2} \\H_{2}^{2} & H_{3}^{2} & H_{4}^{2}\end{bmatrix}\left. {\backslash\lbrack}\begin{matrix}0 \\1 \\0\end{matrix} \right\rbrack}}{v_{A\; 2} = {\left( {v_{1}A_{1}} \right)^{2} + \left( {v_{2}A_{2}} \right)^{2} + \left( {v_{3}\; A_{3}} \right)^{2}}}{v_{AB} = {\left( {v_{1}A_{1}B_{1}} \right)^{2} + \left( {v_{2}A_{2}B_{2}} \right)^{2} + \left( {v_{3}A_{3}B_{3}} \right)^{2}}}{v_{B\; 2} = {\left( {v_{1}B_{1}} \right)^{2} + \left( {v_{2}B_{2}} \right)^{2} + \left( {v_{3}B_{3}} \right)^{2}}}{w_{A\; 2} = {\left( {w_{1}A_{1}} \right)^{2} + \left( {w_{2}A_{2}} \right)^{2} + \left( {w_{3}A_{3}} \right)^{2}}}{w_{AB} = {\left( {w_{1}A_{1}B_{1}} \right)^{2} + \left( {w_{2}A_{2}B_{2}} \right)^{2} + \left( {w_{3}A_{3}B_{3}} \right)^{2}}}{w_{B\; 2} = {\left( {w_{1}B_{1}} \right)^{2} + \left( {w_{2}B_{2}} \right)^{2} + \left( {w_{3}B_{3}} \right)^{2}}}} & (5)\end{matrix}$

Using the parameters defined in equations (1)-(5), solving for θ isaccomplished by solving the eighth-degree polynomial equation (6) for t,where the coefficients Q_(i)(for i=0 to 8) are defined as shown inequations (7)-(15).

0=Q ₈ t ⁸ +Q ₇ t ⁷ +Q ₆ t ⁶ +Q ₅ t ⁵ +Q ₄ t ⁴ +Q ₃ t ³ +Q ₂ t ² +Q ₁ t+Q₀  (6)

The parameters A_(l), B₁, G₁, H₁, v_(A2), v_(AB), v_(B2), w_(A2),w_(AB), and w_(B2) used in equations (7)-(15) are defined as shown inequations (1)-(4). The parameter n is the assumed semi-major axis (inother words, a₀). Once the real roots t are known, the possible valuesof θ are defined as θ=a tan(t).

Q ₈=4A _(j) ² n ² v ² _(B2)+4v _(B2) B ₁ ²(1−n ² v _(A2))−(G ₁(1−n ² v_(A2))W _(B2) +n ² v _(B2) w _(A2)+2H ₁ v _(B2))²  (7)

Q ₇=−(2(2n ² v _(AB)+4H ₁ v _(AB)+2G ₁ n ² v _(AB) w ²+2G ₁(1-n ² v_(AB)))(G ₁(1-n ² v _(A2))w _(B2) +n ² v _(B2) w _(A2)+2H ₁ v _(B2))−8A₁ B ₁ n ² v _(B2) ²+16A ₁ ² n ² v _(AB) v _(B2)+(4(2A ₁ B ₁ ² n ² v_(A2))+2B ₁ ² n ² v _(AB)))v _(B2)+8B ₁ ²(1−n ² v _(A2))v _(AB)  (8)

Q ₆=−(2(2H ₁ v _(B2)+2H ₁ v _(A2) +n ² v _(A2) w _(A2) +n ² v _(B2)(−2w_(AB) +w _(B2))+G ₁(n ² v _(B2)+1)w _(B2)+4G ₁ n ² v _(B2)+1)w _(B2)+4G₁ n ² v _(AB) w _(AB) +G ₁(1−N ² v _(A2))v _(A2)))×(G ₁(1−n ² v _(A2))w_(B2) +n ² v _(B2) w _(A2)+2H ₁ v _(B2))−(2n ² v _(AB) w _(A2)+4H ₁ v_(AB)+2G ₁ n ² v _(AB) w _(B2)+2G ₁(1−n ² v _(A2))w _(AB))²+4B ₁ ² n ² v_(B2) ²−32A ₁ B ₁ n ² v _(AB) v _(B2)+4A ₁ ² n ²(2v _(A2) v _(B2)+4v ²_(AB))+4A ₁ ² n ² v _(B2) ²+(4(A ₁ ²(1−n ² v _(A2))+4A ₁ B ₁ n ² v _(AB)+B ₁ ²(−n ² v _(B2)+1)+B ₁ ²(1−n ² v _(A2))))v _(B2)+(8(2A ₁ B ₁(1−n ² v_(A2))+2B ₁ ₂ n ² v _(AB)))v _(AB)+4B ₁ ²(1−n ² v _(A2))v _(A2)  (9)

Q ₅=−(2(4H ₁ v _(AB)+2G ₁(−n ² v _(B2)+1)w _(AB)+2G ₁ n ² v _(AB) v_(A2)+2n ² v _(A)(−2w _(AB) +w _(B2))))(G ₁(1−n ² v _(A2))w _(B2) +n ² v_(B2) w _(A2)+2H ₁ v _(B2))−(2(2H ₁ v _(B2)+2H ₁ v _(A2) +n ² v _(A2) w_(A2) +n ² v _(B2)(−2w _(AB) +w _(B2))+G ₁(−n ² v _(B2)+1)w _(B2)+4G ₁ n² v _(AB) w _(AB) +G ₁(1−n ² v _(A2))v _(A2)))×(2n ² v _(AB) w _(A2)+4H₁ v _(AB)+2G ₁ n ² v _(AB) w _(B2)+2G ₁(1−n ² v _(A2))w _(AB))+16B ₁ ² n² v _(AB) v _(B2)−8A ₁ B ₁ n ²(2v _(A2) v _(B2)+4v _(AB) ²)+16A ₁ ² n ²v _(A2) v _(AB)−8A ₁ B ₁ n ² v _(B2) ²+16A ₁ ² n ² v _(AB) v _(B2)+(4(2A₁ ² n ² v _(AB)+2A ₁ B ₁(−n ² v _(B2)+1)+2A ₁ B ₁(1−n ² v _(A2))+2B ₁ ²n ² v _(AB)))v _(B2)+(8(A ₁ ²(1−n ² v _(A2))+4A ₁ B ₁ n ² v _(AB) +B ₁²(−n ² v _(B2)+1)+B ₁ ²(1−n ² v _(A2))))v _(AB)+(4(2A ₁ B ₁(1−n ² v_(A2))+2B ₁ ² n ² v _(AB)))v _(A2)  (10)

Q ₄=(4(A ₁ ²(−n ² v _(B2))+A ₁ ²(1−n ² v _(A2))+4A ₁ B ₁ n ² v _(AB) +B₁ ²(−n ² v _(B2)+1)))v _(B2)+(8(2A ₁ ² n ² v _(AB)+2A ₁ B ₁(−n ² v_(B2)+1)+2A ₁ B ₁(1−n ² v _(A2))+2B ₁ ² n ² v _(AB)))v _(AB)+(4(A ₁²(1−n ² v _(A2))+4A ₁ B ₁ n ² v _(AB) +B ₁ ²(−n ² v _(B2)+1)+B ₁ ²(1−n ²v _(A2))))v _(A2)+4B ₁ ² n ²(2v _(A2) v _(B2)+4v _(AB)2)−32A ₁ B ₁ n ² v_(A2) v _(AB)+4A ₁ ² n ² v _(A2) ²+4B ₁ ² n ² v _(B2) ²−32A ₁ B ₁ n ² v_(AB) v _(B2)+4A ₁ ² n ²(2v _(A2) v _(B2)+4v _(AB) ²)−(2(G ₁(−n ² v_(B2)+1)v _(A2) +n ² v _(A2)(−2w _(AB) +w _(B2))+2H ₁ v _(A2)))(G ₁(1−n² v _(A2))w _(B2) +n ² v _(B2) w _(A2)+2H ₁ v _(B2))−(2(4H ₁ v _(AB)+2G₁(−n ² v _(B2)+1)w _(AB)+2G ₁ n ² v _(AB) v _(A2)+2n ² v _(AB)(−2w _(AB)+w _(B2))))×(2n ² v _(AB) w _(A2)+4H ₁ v _(AB)+2G ₁ n ² v _(AB) w_(B2)+2G ₁(1−n ² v _(A2))w _(AB))−(2H ₁ v _(B2)+2H ₁ v _(A2) +n ² v_(A2) w _(A2) +n ² v _(B2)(−2w _(AB) +w _(B2))+G ₁(−n ² v _(B2)+1)w_(B2)+4G ₁ n ² v _(AB) w _(AB) +G ₁(1−n ² v _(A2))v _(A2))²  (11)

Q ₃=−(2G ₁(−n ² v _(B2)+1)v _(A2) +n ² v _(A2)(−2w _(AB) +w _(B2))+2H ₁v _(A2)))(2n ² v _(AB) w _(A2)+4H ₁ v _(AB)+2G ₁ n ² v _(AB) w _(B2)+2G₁(1−n ² v _(A2))w _(AB))−(2(4H ₁ v _(AB)+2G ₁(−n ² v _(B2)+1)w _(AB)+2G₁ n ² v _(AB) v _(A2)+2n ² v _(AB)(−2w _(AB) +w _(B2))))×(2H ₁ v_(B2)+2H ₁ v _(A2) +n ² v _(A2) w _(A2) +n ² v _(B2)(−2w _(AB) +w_(B2))+G ₁(−n ² v _(B2)+1)w _(B2)+4G ₁ n ² v _(AB) w _(AB) +G ₁(1−n ² v_(A2))v _(A2))+16B ₁ ² n ² v _(A2) v _(AB)−8A ₁ B ₁ n ² v _(A2) ²+16B ₁² n ² v _(AB) v _(B2)−8A ₁ B ₁ n ²(2v _(A2) v _(B2)+4_(AB) ²)+16A ₁ ² n² v _(A2) v _(AB)+(4(2A ₁ ² n ² v _(AB)+2A ₁ B ₁(−n ² v _(B2)+1)))v_(B2)+(8(A ₁ ²(−n ² v _(B2)+1)+A ₁ ²(1−n ² v _(A2))+4A ₁ B ₁ n ² v _(AB)+B ₁(−² n ² v _(B2)+1)))v _(AB)+(4(2A ₁ ² n ² v _(AB)+2A ₁ B ₁(−n ² v_(B2)+1)+2A ₁ B ₁(1−n ² v _(A2))+2B ₁ ² n ² v _(AB)))v _(A2)  (12)

Q ₂=4A ₁ ²(−n ² v _(B2)+1)v _(B2)+(8(2A ₁ ² n ² v _(AB)+2A ₁ B ₁(−n ² v_(B2)+1)))v _(AB)+(4(A ₁ ²(−n ² v _(B2)+1)+A ₁ ²(1−n ² v _(A2))+4A ₁ B ₁n ² v _(AB) +B ₁ ²(−n ² v _(B2)+1)))v _(A2)+4B ₁ ² n ² v _(A2) ²+4B ₁ ²n ²(2v _(A2) v _(B2)+4v _(AB) ²)−32A ₁ B ₁ n ² v _(A2) v _(AB)+4A ₁ ² n² v _(A2) ²−(2(G ₁(−n ² v _(B2)+1)v _(A2) +n ² v _(A2)(−2w _(AB) +w_(B2))+2H ₁ v _(A2)))×(2H ₁ v _(B2)+2H ₁ v _(A2) +n ² v _(A2) w _(A2) +n² v _(B2)(−2w _(AB) +w _(B2))+G ₁(−n ² v _(B2)+1)w _(B2)+4G ₁ n ² v_(AB) w _(AB) +G ₁(1−n ² v _(A2))v _(A2))−(4H ₁ v _(AB)+2G ₁(−n ² v_(B2)+1)w _(AB)+2G ₁ n ² v _(AB) v _(A2)+2n ² v _(AB)(−2w _(AB) +w_(B2)))²  (13)

Q ₁=8A ₁ ²(−n ² v _(B2)+1)v _(AB)+(4(2A ₁ ² n ² v _(AB)+2A ₁ B ₁(−n ² v_(B2)+1)))v _(A2)+16B ₁ ² n ² v _(A2) v _(AB)−8A ₁ B ₁ n ² v ²_(A2)−(2(G ₁(−n ² v _(B2)+1)v _(A2) +n ² v _(A2)(−2w _(AB) +w _(B2))+2H₁ v _(A2)))4H ₁ v _(AB)+2G ₁(−n ² v _(B2)+1)w _(AB)+2G ₁ n ² v_(AB)v_(A2)+2n ² v _(AB)(−2w _(AB) +w _(B2)))  (14)

Q ₀=4A ₁ ²(−n ² v _(B2)+1)v _(A2)−(G ₁(−n ² v _(B2)+1)v _(A2) +n ² v_(A2)(−2w _(AB) +w _(B2))+2H ₁ v _(A2))²+4B ₁ ² n ² v ² _(A2)  (15)

In this example embodiment, equations (6)-(15) have at most three realroots; thus, for any four tangent lines, there are at most threepossible ellipses that are tangent to all four lines and that satisfythe a=a₀ constraint. (In some instances, there may be fewer than threereal roots.) For each real root θ, the corresponding values of (x_(C),y_(C)) and b can be readily determined. Depending on the particularinputs, zero or more solutions will be obtained; for example, in someinstances, three solutions can be obtained for a typical configurationof tangents. Each solution is completely characterized by the parameters{θ, a=a₀, b, (x_(C), Y_(C))}. Alternatively, or additionally, a modelbuilder 197C and model updater 197D provide (FIG. 8A) functionality todefine, build and/or customize model(s) 197B using one or morecomponents in object library 197A. Once built, model refiner 197Fupdates and refines the model, bringing the predictive information ofthe model in line with observed information from the detection system90A.

The model subcomponents 197-1, 197-2, 197-3, and 197-4 can be scaled,sized, selected, rotated, translated, moved, or otherwise re-ordered toenable portions of the model corresponding to the virtual surface(s) toconform within the points 193 in space. Model refiner 197F employs avariation detector 197G to substantially continuously determinedifferences between sensed information and predictive information andprovide to model refiner 197F a variance useful to adjust the model 197Baccordingly. Variation detector 197G and model refiner 197F are furtherenabled to correlate among model portions to preserve continuity withcharacteristic information of a corresponding object being modeled,continuity in motion, and/or continuity in deformation, conformationand/or torsional rotations.

In an embodiment, when the control object morphs, conforms, and/ortranslates, motion information reflecting such motion(s) is includedinto the observed information. Points in space can be recomputed basedon the new observation information. The model subcomponents can bescaled, sized, selected, rotated, translated, moved, or otherwisere-ordered to enable portions of the model corresponding to the virtualsurface(s) to conform within the set of points in space.

In an embodiment, motion(s) of the control object can be rigidtransformation, in which case, points on the virtual surface(s) remainat the same distance(s) from one another through the motion. Motion(s)can be non-rigid transformations, in which points on the virtualsurface(s) can vary in distance(s) from one another during the motion.In an embodiment, observation information can be used to adjust (and/orrecomputed) predictive information thereby enabling “tracking” thecontrol object. In embodiments, control object can be tracked bydetermining whether a rigid transformation or a non-rigid transformationoccurs. In an embodiment, when a rigid transformation occurs, atransformation matrix is applied to each point of the model uniformly.Otherwise, when a non-rigid transformation occurs, an error indicationcan be determined, and an error minimization technique such as describedherein above can be applied. In an embodiment, rigid transformationsand/or non-rigid transformations can be composed. One examplecomposition embodiment includes applying a rigid transformation topredictive information. Then an error indication can be determined, andan error minimization technique such as described herein above can beapplied. In an embodiment, determining a transformation can includecalculating a rotation matrix that provides a reduced RMSD (root meansquared deviation) between two paired sets of points. One embodiment caninclude using Kabsch Algorithm to produce a rotation matrix. In anembodiment and by way of example, one or more force lines can bedetermined from one or more portions of a virtual surface.

Collisions

In an embodiment, predictive information can include collisioninformation concerning two or more capsoloids. By means of illustration,several possible fits of predicted information to observed informationcan be removed from consideration based upon a determination that thesepotential solutions would result in collisions of capsoloids. In anembodiment, a relationship between neighboring capsoloids, each havingone or more attributes (e.g., determined minima and/or maxima ofintersection angles between capsoloids) can be determined. In anembodiment, determining a relationship between a first capsoloid havinga first set of attributes and a second capsoloid having a second set ofattributes includes detecting and resolving conflicts between firstattribute and second attributes. For example, a conflict can include acapsoloid having one type of angle value with a neighbor having a secondtype of angle value incompatible with the first type of angle value.Attempts to attach a capsoloid with a neighboring capsoloid havingattributes such that the combination will exceed what is allowed in theobserved—or to pair incompatible angles, lengths, shapes, or other suchattributes can be removed from the predicted information without furtherconsideration.

Lean Model

In an embodiment, predictive information can be artificially constrainedto capsoloids positioned in a subset of the observed information—therebyenabling creation of a “lean model”. For example, as illustrated in FIG.8B, capsoloid 197-3 could be used to denote the portion of the observedwithout addition of capsoloids 197-2. In a yet further embodiment,connections can be made using artificial constructs to link togethercapsoloids of a lean model. In another embodiment, the predictiveinformation can be constrained to a subset of topological informationabout the observed information representing the control object to form alean model. In an embodiment, a lean model can be associated with a fullpredictive model. The lean model (or topological information, orproperties described above) can be extracted from the predictive modelto form a constraint. Then, the constraint can be imposed on thepredictive information thereby enabling the predictive information to beconstrained in one or more of behavior, shape, total (system) energy,structure, orientation, compression, shear, torsion, other properties,and/or combinations thereof.

Occlusions

In an embodiment, the observed can include components reflectingportions of the control object which are occluded from view of thedevice (“occlusions” or “occluded components”). In one embodiment, thepredictive information can be “fit” to the observed as described hereinabove with the additional constraint(s) that some total property of thepredictive information (e.g., potential energy) be minimized ormaximized (or driven to lower or higher value(s) through iteration orsolution). Properties can be derived from nature, properties of thecontrol object being viewed, others, and/or combinations thereof. Inanother embodiment, as shown by FIGS. 8B and 8C, a deformation of thepredictive information subcomponent 359 can be allowed subject to anoverall permitted value of compression, deformation, flexibility,others, and/or combinations thereof.

Friction

In an embodiment, a “friction constraint” is applied on the model197B-1. For example, if fingers of a hand being modeled are closetogether (in position or orientation), corresponding portions of themodel will have more “friction”. The more friction a model subcomponenthas in the model, the less the subcomponent moves in response to newobserved information. Accordingly the model is enabled to mimic the wayportions of the hand that are physically close together move together,and move less overall.

An environmental filter 197H reduces extraneous noise in sensedinformation received from the detection system 90A using environmentalinformation to eliminate extraneous elements from the sensoryinformation. Environmental filter 197H employs contrast enhancement,subtraction of a difference image from an image, software filtering, andbackground subtraction (using background information provided by objectsof interest determiner 198H (see below) to enable model refiner 197F tobuild, refine, manage and maintain model(s) 197B of objects of interestfrom which control inputs can be determined.

A model analyzer 197I determines that a reconstructed shape of a sensedobject portion matches an object model in an object library; andinterprets the reconstructed shape (and/or variations thereon) as userinput. Model analyzer 197I provides output in the form of object,position, motion and attribute information to an interaction system 90C.

Again with reference to FIG. 8A, an interaction system 90C includes aninteraction interpretation module 198 that provides functionality torecognize command and other information from object, position, motionand attribute information obtained from variation system 90B. Aninteraction interpretation module 198 embodiment comprises a recognitionengine 198F to recognize command information such as command inputs(i.e., gestures and/or other command inputs (e.g., speech, etc.)),related information (i.e., biometrics), environmental information (i.e.,context, noise, etc.) and other information discernable from the object,position, motion and attribute information that might be useful incontrolling a machine. Recognition engine 198F employs gestureproperties 198A (e.g., path, velocity, acceleration, etc.), controlobjects determined from the object, position, motion and attributeinformation by an objects of interest determiner 198H and optionally oneor more virtual constructs 198B (see e.g., FIGS. 8D, 8E: 198B-1, 198B-2)to recognize variations in control object presence or motion indicatingcommand information, related information, environmental information andother information discernable from the object, position, motion andattribute information that might be useful in controlling a machine.With reference to FIGS. 8D, 8E, virtual construct 198B-1, 198B-2implement an engagement target with which a control object 99interacts—enabling MSCS 189 to discern variations in control object(i.e., motions into, out of or relative to virtual construct 198B) asindicating control or other useful information. A gesture trainer 198Cand gesture properties extractor 198D provide functionality to define,build and/or customize gesture properties 198A.

A context determiner 198G and object of interest determiner 198H providefunctionality to determine from the object, position, motion andattribute information objects of interest (e.g., control objects, orother objects to be modeled and analyzed), objects not of interest(e.g., background) based upon a detected context. For example, when thecontext is determined to be an identification context, a human face willbe determined to be an object of interest to the system and will bedetermined to be a control object. On the other hand, when the contextis determined to be a fingertip control context, the finger tips will bedetermined to be object(s) of interest and will be determined to be acontrol objects whereas the user's face will be determined not to be anobject of interest (i.e., background). Further, when the context isdetermined to be a styli (or other tool) held in the fingers of theuser, the tool tip will be determined to be object of interest and acontrol object whereas the user's fingertips might be determined not tobe objects of interest (i.e., background). Background objects can beincluded in the environmental information provided to environmentalfilter 197H of model management module 197.

A virtual environment manager 198E provides creation, selection,modification and de-selection of one or more virtual constructs 198B(see FIGS. 8D, 8E). In some embodiments, virtual constructs (e.g., avirtual object defined in space; such that variations in real objectsrelative to the virtual construct, when detected, can be interpreted forcontrol or other purposes (see FIGS. 8D, 8E)) are used to determinevariations (i.e., virtual “contact” with the virtual construct, breakingof virtual contact, motion relative to a construct portion, etc.) to beinterpreted as engagements, dis-engagements, motions relative to theconstruct(s), and so forth, enabling the system to interpret pinches,pokes and grabs, and so forth. Interaction interpretation module 198provides as output the command information, related information andother information discernable from the object, position, motion andattribute information that might be useful in controlling a machine fromrecognition engine 198F to an application control system 90D. Furtherwith reference to FIG. 8A, an application control system 90D includes acontrol module 199 that provides functionality to determine andauthorize commands based upon the command and other information obtainedfrom interaction system 90C.

A control module 199 embodiment comprises a command engine 199F todetermine whether to issue command(s) and what command(s) to issue basedupon the command information, related information and other informationdiscernable from the object, position, motion and attribute information,as received from an interaction interpretation module 198. Commandengine 199F employs command/control repository 199A (e.g., applicationcommands, OS commands, commands to MSCS, misc. commands) and relatedinformation indicating context received from the interactioninterpretation module 198 to determine one or more commandscorresponding to the gestures, context, etc. indicated by the commandinformation. For example, engagement gestures can be mapped to one ormore controls, or a control-less screen location, of a presentationdevice associated with a machine under control. Controls can includeimbedded controls (e.g., sliders, buttons, and other control objects inan application), or environmental level controls (e.g., windowingcontrols, scrolls within a window, and other controls affecting thecontrol environment). In embodiments, controls may be displayed using 2Dpresentations (e.g., a cursor, cross-hairs, icon, graphicalrepresentation of the control object, or other displayable object) ondisplay screens and/or presented in 3D forms using holography,projectors or other mechanisms for creating 3D presentations, or audible(e.g., mapped to sounds, or other mechanisms for conveying audibleinformation) and/or touchable via haptic techniques.

Further, an authorization engine 199G employs biometric profiles 199B(e.g., users, identification information, privileges, etc.) andbiometric information received from the interaction interpretationmodule 198 to determine whether commands and/or controls determined bythe command engine 199F are authorized. A command builder 199C andbiometric profile builder 199D provide functionality to define, buildand/or customize command/control repository 199A and biometric profiles199B.

Selected authorized commands are provided to machine(s) under control(i.e., “client”) via interface layer 196. Commands/controls to thevirtual environment (i.e., interaction control) are provided to virtualenvironment manager 198E. Commands/controls to the emission/detectionsystems (i.e., sensory control) are provided to emission module 91and/or detection module 92 as appropriate.

In various embodiments and with reference to FIGS. 8D, 8E, a MachineSensory Controller System 189 can be embodied as a standalone unit(s)189-1 coupleable via an interface (e.g., wired or wireless)), embedded(e.g., within a machine 188-1, 188-2 or machinery under control) (e.g.,FIG. 8D: 189-2, 189-3, FIG. 8E: 189B) or combinations thereof.

FIG. 9 illustrates a sensory augmentation system to add simulatedsensory information to a virtual reality input. The system is adapted toreceive a virtual reality input including a primitive (901). Virtualreality primitives can include e.g., virtual character, virtualenvironment, others, or properties thereof. The primitive is simulatedby a service side simulation engine (902). Information about a physicalenvironment is sensed and analyzed (905). See also FIGS. 7A, 7B, 7C, 8A,8D, and 8E. A predictive information (e.g., model, etc.) is rendered inan internal simulation engine (906). Predictive information andprocesses for rendering predictive models are described in furtherdetail with reference to FIGS. 8B, 8C. Hands and/or other object typesare simulated (903) based upon results of the object primitivesimulation in the service side simulation engine and the results of theprediction information rendered in an internal simulation engine. (Seealso FIG. 8A: 197I). In embodiments, various simulation mechanisms910-920 are employed alone or in conjunction with one another as well asother existing/emerging simulation mechanisms and/or some combinationthereof can also be utilized in accordance with the requirements of aparticular implementation. The service returns as a result a subset ofobject primitive properties to the client (904). Object primitiveproperties can be determined from the simulation mechanisms 910-920, thepredictive information, or combinations thereof.

In an embodiment, a simulation mechanism comprises simulating the effectof a force (914). In an embodiment, a simulation mechanism comprisesminimizing a cost function (912).

In an embodiment, a simulation mechanism comprises detecting a collision(910).

In an embodiment, a simulation mechanism comprises determining a meaningin context (916). Sometimes, determining a meaning in context furthercomprises eye tracking In some applications determining a meaning incontext further comprises recognizing at least one parameter of thehuman voice.

In an embodiment, a simulation mechanism comprises recognizing an objectproperty dependence (e.g., understanding how scale and orientation ofprimitive affects interaction.

In an embodiment, a simulation mechanism comprises vector or tensormechanics (920).

FIG. 10 illustrates an example computing system 1000, such as a PC (orother suitable “processing” system), that can comprise one or more ofthe MSCS elements shown in FIGS. 7A, 7B, 7C, 8A, 8D, and 8E according toan embodiment. While other application-specific device/processalternatives might be utilized, such as those already noted, it will bepresumed for clarity sake that systems 90A-90D elements (FIGS. 7A, 7B,7C, 8A, 8D, and 8E) are implemented by one or more processing systemsconsistent therewith, unless otherwise indicated.

As shown, computer system 1000 comprises elements coupled viacommunication channels (e.g. bus 1001) including one or more general orspecial purpose processors 1002, such as a Pentium® or Power PC®,digital signal processor (“DSP”), or other processing. System 1000elements also include one or more input devices 1003 (such as a mouse,keyboard, joystick, microphone, remote control unit, tactile, biometricor other sensors, and so on), and one or more output devices 1004, suchas a suitable display, joystick feedback components, speakers, biometricor other actuators, and so on, in accordance with a particularapplication.

System 1000 elements also include a computer readable storage mediareader 1005 coupled to a computer readable storage medium 1006, such asa storage/memory device or hard or removable storage/memory media;examples are further indicated separately as storage device 1008 andnon-transitory memory 1009, which can include hard disk variants,floppy/compact disk variants, digital versatile disk (“DVD”) variants,smart cards, read only memory, random access memory, cache memory orothers, in accordance with a particular application (e.g. see datastore(s) 197A, 198A, 199A and 199B of FIG. 8A). One or more suitablecommunication devices 1007 can also be included, such as a modem, DSL,infrared, etc. for providing inter-device communication directly or viasuitable private or public networks, such as the Internet. Workingmemory 1009 is further indicated as including an operating system (“OS”)1091, interaction discriminator 1013 and other programs 1092, such asapplication programs, mobile code, data, or other information forimplementing systems 90A-90D elements, which might be stored or loadedtherein during use.

System 1000 element implementations can include hardware, software,firmware or a suitable combination. When implemented in software (e.g.as an application program, object, downloadable, servlet, and so on, inwhole or part), a system 1000 element can be communicated transitionallyor more persistently from local or remote storage to memory forexecution, or another suitable mechanism can be utilized, and elementscan be implemented in compiled, simulated, interpretive or othersuitable forms. Input, intermediate or resulting data or functionalelements can further reside more transitionally or more persistently ina storage media or memory, (e.g. storage device 1008 or memory 1009) inaccordance with a particular application.

Certain potential interaction determination, virtual object selection,authorization issuances and other aspects enabled by input/outputprocessors and other element embodiments disclosed herein can also beprovided in a manner that enables a high degree of broad or even globalapplicability; these can also be suitably implemented at a lowerhardware/software layer. Note, however, that aspects of such elementscan also be more closely linked to a particular application type ormachine, or might benefit from the use of mobile code, among otherconsiderations; a more distributed or loosely coupled correspondence ofsuch elements with OS processes might thus be more desirable in suchcases.

FIG. 11A illustrates a system for capturing image and other sensory dataaccording to an implementation of the technology disclosed.

Refer first to FIG. 11A, which illustrates a system for capturing imagedata according to one implementation of the technology disclosed. System1100 is preferably coupled to a wearable device 1101 that can be apersonal head mounted display (HMD) having a goggle form factor such asshown in FIG. 11A, a helmet form factor, or can be incorporated into orcoupled with a watch, smartphone, or other type of portable device.

In various implementations, the system and method for capturing 3Dmotion of an object as described herein can be integrated with otherapplications, such as a head-mounted device or a mobile device.Referring again to FIG. 11A, a head-mounted device 1101 can include anoptical assembly that displays a surrounding environment or a virtualenvironment to the user; incorporation of the motion-capture system 1100in the head-mounted device 1101 allows the user to interactively controlthe displayed environment. For example, a virtual environment caninclude virtual objects that can be manipulated by the user's handgestures, which are tracked by the motion-capture system 1100. In oneimplementation, the motion-capture system 1100 integrated with thehead-mounted device 1101 detects a position and shape of user's hand andprojects it on the display of the head-mounted device 1100 such that theuser can see her gestures and interactively control the objects in thevirtual environment. This can be applied in, for example, gaming orInternet browsing.

In one embodiment, information about the interaction with a virtualobject can be shared by a first HMD user with a HMD of a second user.For instance, a team of surgeons can collaborate by sharing with eachother virtual incisions to be performed on a patient. In someembodiments, this is achieved by sending to the second user theinformation about the virtual object, including primitive(s) indicatingat least one of a type, size, and/or features and other informationabout the calculation point(s) used to detect the interaction. In otherembodiments, this is achieved by sending to the second user informationabout the predictive model used to track the interaction.

System 1100 includes any number of cameras 1102, 1104 coupled to sensoryprocessing system 1106. Cameras 1102, 1104 can be any type of camera,including cameras sensitive across the visible spectrum or with enhancedsensitivity to a confined wavelength band (e.g., the infrared (IR) orultraviolet bands); more generally, the term “camera” herein refers toany device (or combination of devices) capable of capturing an image ofan object and representing that image in the form of digital data. Forexample, line sensors or line cameras rather than conventional devicesthat capture a two-dimensional (2D) image can be employed. The term“light” is used generally to connote any electromagnetic radiation,which may or may not be within the visible spectrum, and may bebroadband (e.g., white light) or narrowband (e.g., a single wavelengthor narrow band of wavelengths).

Cameras 1102, 1104 are preferably capable of capturing video images(i.e., successive image frames at a constant rate of at least 15 framesper second); although no particular frame rate is required. Thecapabilities of cameras 1102, 1104 are not critical to the technologydisclosed, and the cameras can vary as to frame rate, image resolution(e.g., pixels per image), color or intensity resolution (e.g., number ofbits of intensity data per pixel), focal length of lenses, depth offield, etc. In general, for a particular application, any camerascapable of focusing on objects within a spatial volume of interest canbe used. For instance, to capture motion of the hand of an otherwisestationary person, the volume of interest might be defined as a cubeapproximately one meter on a side.

As shown, cameras 1102, 1104 can be oriented toward portions of a regionof interest 1112 by motion of the device 1101, in order to view avirtually rendered or virtually augmented view of the region of interest1112 that can include a variety of virtual objects 1116 as well ascontain an object of interest 1114 (in this example, one or more hands)moves within the region of interest 1112. One or more sensors 1108, 1110capture motions of the device 1101. In some implementations, one or morelight sources 1115, 1117 are arranged to illuminate the region ofinterest 1112. In some implementations, one or more of the cameras 1102,1104 are disposed opposite the motion to be detected, e.g., where thehand 1114 is expected to move. This is an optimal location because theamount of information recorded about the hand is proportional to thenumber of pixels it occupies in the camera images, and the hand willoccupy more pixels when the camera's angle with respect to the hand's“pointing direction” is as close to perpendicular as possible. Sensoryprocessing system 1106, which can be, e.g., a computer system, cancontrol the operation of cameras 1102, 1104 to capture images of theregion of interest 1112 and sensors 1108, 1110 to capture motions of thedevice 1101. Information from sensors 1108, 1110 can be applied tomodels of images taken by cameras 1102, 1104 to cancel out the effectsof motions of the device 1101, providing greater accuracy to the virtualexperience rendered by device 1101. Based on the captured images andmotions of the device 1101, sensory processing system 1106 determinesthe position and/or motion of object 1114.

For example, as an action in determining the motion of object 1114,sensory processing system 1106 can determine which pixels of variousimages captured by cameras 1102, 1104 contain portions of object 1114.In some implementations, any pixel in an image can be classified as an“object” pixel or a “background” pixel depending on whether that pixelcontains a portion of object 1114 or not. Object pixels can thus bereadily distinguished from background pixels based on brightness.Further, edges of the object can also be readily detected based ondifferences in brightness between adjacent pixels, allowing the positionof the object within each image to be determined. In someimplementations, the silhouettes of an object are extracted from one ormore images of the object that reveal information about the object asseen from different vantage points. While silhouettes can be obtainedusing a number of different techniques, in some implementations, thesilhouettes are obtained by using cameras to capture images of theobject and analyzing the images to detect object edges. Correlatingobject positions between images from cameras 1102, 1104 and cancellingout captured motions of the device 1101 from sensors 1108, 1110 allowssensory processing system 1106 to determine the location in 3D space ofobject 1114, and analyzing sequences of images allows sensory processingsystem 1106 to reconstruct 3D motion of object 1114 using conventionalmotion algorithms or other techniques. See, e.g., U.S. patentapplication Ser. No. 13/414,485 (filed on Mar. 7, 2012) and U.S.Provisional Patent Application Nos. 61/724,091 (filed on Nov. 8, 2012)and 61/587,554 (filed on Jan. 7, 2012), the entire disclosures of whichare hereby incorporated by reference.

Presentation interface 1120 employs projection techniques in conjunctionwith the sensory based tracking in order to present virtual (orvirtualized real) objects (visual, audio, haptic, and so forth) createdby applications loadable to, or in cooperative implementation with, thedevice 1101 to provide a user of the device with a personal virtualexperience. Projection can include an image or other visualrepresentation of an object.

One implementation uses motion sensors and/or other types of sensorscoupled to a motion-capture system to monitor motions within a realenvironment. A virtual object integrated into an augmented rendering ofa real environment can be projected to a user of a portable device 1101.Motion information of a user body portion can be determined based atleast in part upon sensory information received from imaging 1102, 1104or acoustic or other sensory devices. Control information iscommunicated to a system based in part on a combination of the motion ofthe portable device 1101 and the detected motion of the user determinedfrom the sensory information received from imaging 1102, 1104 oracoustic or other sensory devices. The virtual device experience can beaugmented in some implementations by the addition of haptic, audioand/or other sensory information projectors. For example, an optionalvideo projector 1120 can project an image of a page (e.g., virtualdevice) from a virtual book object superimposed upon a real worldobject, e.g., desk 1116 being displayed to a user via live video feed;thereby creating a virtual device experience of reading an actual book,or an electronic book on a physical e-reader, even though no book nore-reader is present. Optional haptic projector can project the feelingof the texture of the “virtual paper” of the book to the reader'sfinger. Optional audio projector can project the sound of a page turningin response to detecting the reader making a swipe to turn the page.Because it is a virtual reality world, the back side of hand 1114 isprojected to the user, so that the scene looks to the user as if theuser is looking at the user's own hand(s).

A plurality of sensors 1108, 1110 coupled to the sensory processingsystem 1106 to capture motions of the device 1101. Sensors 1108, 1110can be any type of sensor useful for obtaining signals from variousparameters of motion (acceleration, velocity, angular acceleration,angular velocity, position/locations); more generally, the term “motiondetector” herein refers to any device (or combination of devices)capable of converting mechanical motion into an electrical signal. Suchdevices can include, alone or in various combinations, accelerometers,gyroscopes, and magnetometers, and are designed to sense motions throughchanges in orientation, magnetism or gravity. Many types of motionsensors exist and implementation alternatives vary widely.

The illustrated system 1100 can include any of various other sensors notshown in FIG. 11A for clarity, alone or in various combinations, toenhance the virtual experience provided to the user of device 1101. Forexample, in low-light situations where free-form gestures cannot berecognized optically with a sufficient degree of reliability, system1106 may switch to a touch mode in which touch gestures are recognizedbased on acoustic or vibrational sensors. Alternatively, system 1106 mayswitch to the touch mode, or supplement image capture and processingwith touch sensing, when signals from acoustic or vibrational sensorsare sensed. In still another operational mode, a tap or touch gesturemay act as a “wake up” signal to bring the image and audio analysissystem 1106 from a standby mode to an operational mode. For example, thesystem 1106 may enter the standby mode if optical signals from thecameras 1102, 1104 are absent for longer than a threshold interval.

It will be appreciated that the objects shown in FIG. 11A areillustrative. In some implementations, it may be desirable to house thesystem 1100 in a differently shaped enclosure or integrated within alarger component or assembly. Furthermore, the number and type of imagesensors, motion detectors, illumination sources, and so forth are shownschematically for the clarity, but neither the size nor the number isthe same in all implementations.

Reference in the specification to “one embodiment” or “an embodiment”means that a particular feature, structure, or characteristic describedin connection with the embodiment is included in at least one embodimentof the technology disclosed. The appearances of the phrase “in oneembodiment” in various places in the specification are not necessarilyall referring to the same embodiment.

Spring Zooming Camera Movement

FIGS. 11B, 11C, and 11D illustrate an example of a “spring zooming”camera movement of a virtual camera in a three dimensional (3D) sensorspace. In one implementation, a user can “pull” the virtual camera 1142(as enabled by MSCS), using manipulation points, analytic fits, or othertype of mechanics that emulate spring dynamics. When the user “releases”the camera, the camera can then move as if it were a bob of constantmass on a spring. In other implementations, the camera movement stopsafter a fixed number of oscillations of this virtual spring. In yetanother implementation, such spring movement describes a one-directionalcamera trajectory from one fixed point to another.

FIG. 11B illustrates the configuration of an example “slingshot” in 4frames, where the slingshot moves the camera from a one-directionalcamera trajectory from one fixed point to another. This motion iscalculated by both the radius of the control circle and the “pull back”of the camera. The first frame indicates the creation of a controlcircle 1102, which defines a starting point and strength of the“slingshot”. In frame 2, the user pulls back the camera 1104 to providean input of force to be applied to the camera 1104. In frame 3, releaseof the camera 1104 applies the virtual force to the camera therebymoving the camera to a new position in space. In frame 4, depicted iswhere the radius of the control circle 1106 as well as the distance ofthe pullback 1108 of the camera 1104 identifies the force that will beapplied to the camera 1104 when the camera is released.

FIG. 11C illustrates an alternative example of a slingshot cameramovement implementation in which a virtual spring is employed where thecamera moves as if it were a bob of constant mass on a spring. In aframe 1, a user stroke 1132 defines some contour relative to the camera1134. In one implementation, a circle of best-fit is fit to a pluralityof points on this contour. In frame 2, the radius 1136 of the circle1132 defines at least one parameter of a spring, the spring constant k.The static length of a spring 1 is defined by a length 1138 whichextends from the center of the circle to the current position of thecamera 1134. In this example, the spring constant k 1140 is equal to theradius of the circle 1136.

In frame 3, illustrated is using Hooke's law, which is stated as F=−kXwhere k is the spring constant 1142 and X 1144 is the distance that thespring is extended, the potential energy in the spring after it ispulled back will be equal to 1/2kX². Once the camera is released thispotential energy will move the camera toward the center of the circle.In one implementation the camera will have a given mass, andacceleration will be constant. This allows the calculation of thevelocity of the camera once it is released. In this example the camerawill move along the trajectory defined by the pull back,

FIG. 11D illustrates in frame 4 a slingshot camera movement result wherethe camera is released once it traverses the vector created by the pullback. Once the camera has travelled the distance X 1144 of FIG. 11C,which can be defined as the distance of the camera from the center ofthe circle ‘d’ minus the starting point of the camera ‘1’ 1162 thespring is effectively removed from the force equation, allowing thecamera to move as it would if no other forces were applied. In frame 5of FIG. 11D, the spring constant ‘k’ can be multiplied by a relationshipbetween the spring constant and the radius of the circle, where a largerradius signifies a larger number of springs. For example, a radius of 2can indicate 1 spring, a radius of 4 can indicate 2 springs, and so on.

Camera Selection Controls

FIGS. 12A-12C illustrate defining and controlling multiple virtualcameras in a three dimensional (3D) sensor space. In one implementation,after setting the context to creating a camera, a user defines at leastone point with a direction vector in virtual space 1208. This point andvector defines a “camera selector.” The user can iterate through aplurality of such pre-defined camera controls 1218. In otherimplementations, iteration through checkpoints can be enabled by avisual interaction element, which represents at least one such-definedcamera selector. In one implementation, interactions with cameraselectors can be enabled by vector-space interactions with interactionelements. In some other implementations, cameras can be created ordestroyed by circle gestures around the cameras in virtual space, theirvisual representation, or combinations thereof. In yet otherimplementations, circle gestures can be defined as the circle of bestfit to contours as described herein.

FIG. 12A illustrates checkpoint camera controls in three frames. Aso-called checkpoint sets a camera perspective, including location, viewdirection, focal length of the camera, and a selector box in an array ofselector boxes. For instance, perspective 1208 is defined by a circlegesture in frame 1. The anchor is at the center of the circle. A menu,for instance, allows the user to further define the view direction andthe focal length of a lens position at the center of the circle.Gestures provide alternatives for setting camera controls.

FIG. 12A frame 2 further illustrates using selector boxes 1218 to selectamong the four checkpoints or perspectives created in this example.

FIG. 12A frame 3 further illustrates adding a fifth checkpoint to thefirst four. When the fifth checkpoint is added, represented by a pointand vector, an additional selector box can be provided as part of thearray of selector boxes.

FIG. 12B illustrates the manipulations of the camera in 4 frames. Thefirst frame shows the creation of a circle with a center point and adirection. The second frame shows the creation of additional cameras forthe example. And the third frame demonstrates the selector boxes 1218created as part of an array of selector boxes for this example.

Frame 4 in FIG. 12B illustrates the selection of a camera 1244 via aselection box 1218 for manipulation. In this example, once the camera isselected it can be moved to a new point in space, its focal length canbe modified, its vector can be changed, and any other attributes thathave been assigned to it can be changed by the gestures identified inthe technology disclosed.

FIG. 12C illustrates examples of other manipulations possible using thetechnology disclosed. Frame 5 is an example of selecting and modifying adifferent camera 1262 than the camera selected in FIG. 12B. Frame 6illustrates the creation of a new camera 1272 while camera 2 1262remains as the selected camera.

Frame 7 of FIG. 12C illustrates the removal of a camera 1272. A circleis drawn around the virtual camera 1282 in the opposite direction of thecircle used to create the camera 1272. In this example, its selector box1218 a is deleted from the selector box array 1218.

Pluck and Release Camera Controls

FIG. 13 illustrates in 4 Frames a camera set in motion by pluck andrelease camera controls in a three dimensional (3D) sensor space. InFrame 1 a user defines a central location with reference to a salientproperty of a stroke. In one implementation, this property can be alocation in virtual space of the center of a circle of best fit to thestroke 1308. In another implementation, this point can be the new centerpoint of a camera. In yet other implementations, the user can grab andmanipulate the camera as if it were a bob of uniform mass attached to aspring 1328, as shows in Frame 2. As illustrated in Frame 3, releasingthe camera returns it to the center point 1338. Frame 4 illustrates anexample of the selection of the origin 1348, and its movement to a newlocation 1350.

Sphere Grabbing Camera Manipulation

FIG. 14, in 4 Frames, illustrates a sphere grabbing camera manipulationin a three dimensional (3D) sensor space. A user describes a sphere invirtual space 1408. In one implementation, this sphere is the sphere ofbest fit, which is fit to a plurality of points on one hand. In otherimplementations this sphere is fit to a plurality of points on twohands. In yet further implementations, the sphere is defined in relationto the hand at the initialization of some “sensory” control, for examplesaying to the computer “define sphere” via voice controls, or looking ata predefined space in a predefined way (e.g. via eye-tracking controls);disengagement gestures may be defined likewise.

Once the virtual sphere is described, a camera view is defined inrelation to the sphere. In an implementation, the camera is defined atthe center of the sphere, with view vector extending from the center ofthe sphere to the point on the surface of the sphere that is equidistantto a plurality of points on the user's hand(s). The sphere can berotated, translated, and scaled to corresponds to movements of at leastone camera control 1408, 1418, 1428, 1438. In other implementations,sphere includes any three-dimensional solid that can be fit to aplurality of points input from an MSCS FIGS. 7A, 7B, 7C, 8A, 8D, and 8E.

Path Creation Camera Control

FIG. 15A-15C illustrate path creation camera controls in a threedimensional (3D) sensor space. In frame 1 of FIG. 15A, a user defines acontinuous contour 1508 through time as described above with respect tothe MSCS. In one implementation the contour is defined by the movementof a user's finger through three-dimensional space during a delimitedtime window; in another implementation the contour is defined by themovement of the center of the user's palm. In frame 2, the cameratraverses the defined contour 1508 at a fixed or varying speed from endto end; in frame 3, a user may perform a second gesture, for example aswipe from the left of the screen to the right, to position the cameraat any point on the predefined contour 1528.

In FIG. 15B, movement of a camera along a specified path is shown withreference to an implementation alternative. As depicted in Frame 1 ofFIG. 15B, the user creates a starting point of a path with a creationgesture, e.g., a circle, voice command or other gesture. In frame 2, theuser indicates a path they wish to define by moving their hand (orportion thereof) through 3D space. In frame 3, the user ends the pathwith the same (or different) creation gesture. In frame 4, the cameracan move along the path or 2D object in 3D space.

In FIG. 15C, movement of a camera along a specified path is shown withreference to a yet further implementation alternatives. As depicted inFrame 5 of FIG. 15C, the path is mapped to a straight line that the usercan move easily along, thereby freeing the user from having to trace outthe intricacies of the path in space. As depicted in Frame 6 of FIG.15C, the camera remains attached to the path, but a force is applied inthe direction of the tip of the user's index finger. As depicted inFrame 7 of FIG. 15C, the camera is not rigidly attached to the path, butis subject to a restorative force acting on it (F_(p)). The user isenabled to create a force proportional to the distance to the hand(F_(n)). As depicted in Frame 8 of FIG. 15C, the path is defined by aseries of user created circles (or other shapes) that construct a Beziercurve based on a circle size and vector direction.

User Defined Vortex

FIG. 16 illustrates an implementation where vectors on a control portionof a user's hand interact with vectors in the virtual field. The user isenabled to define the area, direction, strength and velocity of awhirlpool. This interaction creates a field disturbance. In oneimplementation, this field disturbance is a vortex 1602. The interactionof the field disturbance with a vector defined on a virtual objectcreates field disturbance control information. FIG. 16 shows a handcreating a field disturbance by making swirling motions in a virtualspace. Directionality of the swirling motion can control the directionof the whirlpool, and whether objects caught up in the whirlpoolconverge to the center or spin off away from the center. The fielddisturbance interacts with vectors defined on objects in the virtualfield, and moves them once the objects have entered the event horizon ofthe whirlpool. In another implementation, field disturbances themselvescan be manipulated as virtual objects within the space. In oneimplementation, the larger the whirlpool, the more effort the use mustput forth to move it (10 object vortex might require a full handed cupto move, while 2 or 3 object vortex can simply be a flick of a finger orpush of a few fingers.

FIG. 17 illustrates an augmented reality application where a virtualspace 1702 is overlaid on a physical space 1704 to create a syntheticspace 1700. For example, virtual space 1702 includes chess game andtimers 1706 (and/or other game information not shown in FIG. 17 forclarity sake) and is overlaid on physical space 1704, e.g., the table,to provide an augmented reality environment 1700. In one implementation,vectors can be defined on portions of the virtual space, the physicalspace, the synthetic space, or any combination thereof.

An augmented interaction refers to an interaction between vectorsdefined on at least a physical object portion of a user (e.g., the hand)and vectors defined on the synthetic space. In one implementation, anaugmented interaction can modify positional, material, or otherproperties of object portions in synthetic space. In otherimplementations, vectors can be defined to extend outward on the user'sthumb and index finger. Likewise radial vectors can be defined to extendout of the virtual chess pieces. When the vectors of the user's fingertips interact with the vectors of the chess pieces, an augmentedinteraction takes place, as shows in FIG. 17. For example, one or morevectors can indicate potential directions of motion of the selectedchess piece to the user. The hand's motion can be described usingvectors as well. The interaction of the two sets of vectors candetermine whether the user is making a “legal” move and use this derivedinformation to change the presentation from green (ok) to red (illegal).

FIG. 18 illustrates an implementation where an augmented interaction canbe used to navigate a menu system. In FIG. 18, a virtual interactionspace is defined by a user in the physical space and linked to otherimages created by a computer vision. The example shows in FIG. 18 showsa user creating radial-shape “interaction bubbles” with menu elementsinside them 1802, 1804, 1806, which are response to user's gestures. Forexample, the user can select a recipe for a desired dish for dinner frommenu 1802. A display window 1804 pulls up contents of the refrigeratorand pantry, enabling the user to cross reference ingredients needed withfoodstuffs on hand. A menu 1806 can be formed by the user based upon adifference between items in menu 1802 and 1804, generating a shoppinglist that can be downloaded to the user's personal device, uploaded tothe cloud for access later, transmitted to the user's spouse forprocuring on the way home from the office or various combinationsthereof.

Reference in the specification to “one implementation” or “animplementation” means that a particular feature, structure, orcharacteristic described in connection with the implementation isincluded in at least one implementation of the technology disclosed. Theappearances of the phrase “in one implementation” in various places inthe specification are not necessarily all referring to the sameimplementation.

While the technology disclosed has been described by way of example andin terms of the specific implementations, it is to be understood thatthe technology disclosed is not limited to the disclosedimplementations. To the contrary, it is intended to cover variousmodifications and similar arrangements as would be apparent to thoseskilled in the art. Therefore, the scope of the appended claims shouldbe accorded the broadest interpretation so as to encompass all suchmodifications and similar arrangements.

1. A computer implemented method of defining interface modalities in athree dimensional (3D) sensor space, the method including: detecting acontrol gesture of a control object in a three dimensional (3D) sensorspace; calculating gesture parameters of the control gesture that wasdetected; and defining spatial attributes of an interaction modality inthe 3D sensor space responsive to the gesture parameters of the controlgesture.
 2. The method of claim 1, wherein the gesture parametersinclude at least length and width of the control gesture.
 3. The methodof claim 1, wherein the gesture parameters include at least structure,scale, orientation, or density of the control object.
 4. The method ofclaim 1, wherein the spatial attributes include at least height andwidth of an interaction space.
 5. The method of claim 1, wherein thespatial attributes include at least numerosity of elements in theinteraction modality.
 6. The method of claim 1, further includingdetecting a context-setting control gesture that identifies a contextfor interpreting a subsequent control gesture that defines spatialattributes of the interaction modality.
 7. The method of claim 6,wherein the context-setting control gesture is a voice, visual, ordevice command.
 8. The method of claim 6, wherein the subsequent controlgesture applies to an entire interaction space.
 9. The method of claim6, wherein the subsequent control gesture applies to an element of theinteraction space.
 10. The method of claim 6, further including creatingcontext-aware elements of the interaction modality that automaticallyinterpret the context-setting control gesture and subsequent controlgesture to define spatial attributes of the interaction modality. 11.The method of claim 1, wherein the control gesture is a stroke of a userappendage.
 12. The method of claim 1, wherein the control object is adetectable object and the control gesture defines a collection ofcontinuous points that have at least one parameter in common within athreshold deviation.
 13. The method of claim 1, wherein a thresholddeviation is determined by a variation in angle along velocity vectorsthat are continuous in time.
 14. The method of claim 1, wherein thecontrol gesture is a circular sweep that defines a collection of pointswithin a radial distance to a fixed point.
 15. A computer implementedmethod of creating a user-defined virtual interaction modality in athree dimensional (3D) sensor space, the method including: detecting avertical sweep of a control object responsive to a first control gesturein a three dimensional (3D) sensor space; defining a vertical extent ofa virtual interaction space in proportion to length of vertical sweep ofthe control object; detecting a horizontal sweep of the control objectresponsive to a second control gesture in the 3D sensor space; defininga horizontal extent of the virtual interaction space in proportion towidth of horizontal sweep of the control object; and manipulatingcontrols in a physical interaction space by superimposing the virtualinteraction space on the physical interaction space responsive to thevertical extent and horizontal extent.
 16. A computer implemented methodof creating a user-defined virtual interaction modality in a threedimensional (3D) sensor space, the method including: detecting avertical sweep of a control object responsive to a first control gesturein a three dimensional (3D) sensor space; defining a vertical extent ofa virtual interaction space in proportion to length of vertical sweep ofthe control object; detecting a horizontal sweep of the control objectresponsive to a second control gesture in the 3D sensor space; defininga horizontal extent of the virtual interaction space in proportion towidth of horizontal sweep of the control object; and manipulatingcontrols in a synthetic interaction space by linking the virtualinteraction space to an image responsive to the vertical extent andhorizontal extent.
 17. A computer implemented method of creating auser-defined virtual interaction modality in a three dimensional (3D)sensor space, the method including: detecting a circular sweep of acontrol object responsive to a control gesture in a three dimensional(3D) sensor space; calculating a radius of the circular sweep based on afound point that is equidistant to a plurality of points defined oncontour of the control gesture; constructing a radial-based virtualinteraction modality in the 3D sensor space that is in proportion to theradius of the circular sweep; and manipulating controls in a physicalinteraction space by superimposing the radial-based virtual interactionmodality on the physical interaction space responsive to the circularsweep.
 18. A computer implemented method of creating a user-definedvirtual interaction modality in a three dimensional (3D) sensor space,the method including: detecting a circular sweep of a control objectresponsive to a control gesture in a three dimensional (3D) sensorspace; calculating a radius of the circular sweep based on a found pointthat is equidistant to a plurality of points defined on contour of thecontrol gesture; constructing a radial-based virtual interactionmodality in the 3D sensor space that is in proportion to the radius ofthe circular sweep; and manipulating controls in a synthetic interactionspace by linking the radial-based virtual interaction modality to animage responsive to the vertical extent and horizontal extent.
 19. Acomputer implemented method of creating a user-defined virtualinteraction modality in a three dimensional (3D) sensor space, themethod including: identifying a pair of starting points in respectivecenters of two control objects that are detected in a three dimensional(3D) sensor space, wherein the pair of starting points are fixeddistance apart; detecting an outward expanding movement of the controlobjects in the 3D sensor space; identifying a pair of resting points inrespective centers of the two control objects when the control objectscome to rest; defining a horizontal extent of a virtual interactionspace in proportion to distance between the starting points and theresting points; defining a vertical extent of the virtual interactionspace in proportion to width of the control objects; and presenting theinteraction space responsive to the vertical extent and horizontalextent.
 20. The method of claim 19, wherein the two control objects aretwo user appendages.
 21. A computer implemented method of creating auser-defined virtual interaction modality in a three dimensional (3D)sensor space, the method including: identifying a pair of startingpoints in respective centers of control points of one or more controlobjects that are detected in a three dimensional (3D) sensor space,wherein the pair of starting points are fixed distance apart; detectingan outward expanding movement of the control points in the 3D sensorspace; identifying a pair of resting points in respective centers of thecontrol points when the control points come to rest; defining ahorizontal extent of a virtual interaction space in proportion todistance between the starting points and the resting points; defining avertical extent of the virtual interaction space in proportion to widthof the control objects; and presenting the interaction space responsiveto the vertical extent and horizontal extent.
 22. The method of claim21, wherein the control objects are hands and control points are fingertips.